The app
I’ve spent some recent spare time putting together a Bike Price Guide web app. I use the eBay API to scrape listings, then serve the completed listings in a searchable UI.
It can help when pricing out a bike or components you may be interested in buying or selling.
I took motivation for this from a few similar sites that exist for vinyl record auctions (Gripsweat, Popsike, or Roots Vinyl Guide). The first two sell subscription services, the final one serves ads. Mine does neither (yet?).
The tech
There are two pieces to this app:
- Scraping data from eBay.
- Serving the web app.
Scraping data from eBay
There is no easy way to query eBay’s API for completed items directly. The findCompletedItems endpoint is deprecated, and the replacement Marketplace Insights API is restricted. So, we first have to query for live listings, wait for those to expire, and then query the listings again after completion.
I’ve got two python scripts running for this, both are running as cron jobs on my DigitalOcean droplet.
The first uses eBay’s FindingService to load the most recently added items. I query recently added items so that I can capture fixed price (“Buy It Now”) items that may sell quickly after listing. I store the item ID and ending timestamps returned from the FindingService in a sqlite3 table. The second script queries that table for items that have completed, and then queries the TradingService for the completed item’s information. It stores the results of each item, including all images in the same sqlite3 db.
The output of the scraping system is a single sqlite3db with three tables, one for items that we are waiting to complete, one for items that have completed, and one for images from the completed items.
Serving the web app
I serve the webapp using the same database populated by our scraping system.
The app is served using Flask with jinja2 templating. I haven’t added any css styling, and the only javascript is Google Analytics.
I run the Flask app on my DigitalOcean droplet using uWSGI.
Lessons learned
Aside from this blog, this is the first project outside of my real work in a long time that I’ve shipped and made public to any degree. So, that’s fun.
Some quick lessons learned / rehashed:
- Sqlite3 is just plain awesome. The entire bike price guide is served from 1 python file, 3 html templates, and 1 sqlite3 db. The db contains all eBay items and all images. I expect it to scale without issue in terms of number of items in the db. If the app ever gets non-trivial traffic, or if I want to serve from more than a single droplet instance, I may have some other considerations.
- Just write the tests. I wouldn’t write untested code at work, but for fun hobby projects? YOLO! In reality, I made a few silly errors because of it. Nothing that really caused issues, but one or two simple tests on the endpoints would have saved a few faulty commits.
- Ship the thing. A goal with this blog and these projects is to build, ship, and move. Sure this app has some room for improvement and I could nit the code, but it was a quick project and it’s time to mooooove on.
What’s next
I’m going to let this run for a while. It doesn’t really add any value until it’s been collecting items for longer than 3 months. At that point, the app will be serving older data than is available directly through eBay.
I’ll also likely pick another category and run a second app for funsies.
Eventually, I may decide to move it over to its own domain, give it some styling, and monetize it.
For now though, it was a fun little side project, and I’m going to put it aside to move onto the next project!