I've meant to build this for months but never got around to it until this weekend.
Basically, out of sheer curiosity, I want to know the fastest-growing Meetups in Berlin. I want to keep a finger on the pulse of this city. For that I find that Meetup memberships are one clear, physical indicator of what people are subscribing to. And what they consider worth their time to eventually meet up for.
So, anyways, I came up with a system that scrapes Berlin's Meetups every day and calculates the member growth-rate in percent. It allows me to look at the daily report via a simple web UI:
First, the fastest growing Meetups per day...
...and yesterday's Meetups compared and sorted:
I also added a simple page for each Meetup that summarizes its growth-journey:
## A few things I learned
### The Meetup API gives me varying responses
The Meetup API responses vary in their results – probably because I'm doing something wrong.
Basically, for Berlin, I have to page through 26 pages of results with 100 Meetups each. That's about 2,660 Meetups in Berlin.
When I run my job once a day, sometimes I only get 1 page, sometimes 10, sometimes 14 and very rarely the full 26 pages. The API just eventually responds with an empty array on which I finish. I always get 200 OK responses, too, which makes it harder to understand why the variation occurs.
To better understand the issue, I've created a monitoring table. If my job isn't able to fetch the latest data for all 2,660+ Meetups I already know exist, the job is marked as "Incomplete".
As you can see, two days ago, I fetched all Meetups while the next time I ran the job only 1,400 (14 pages with 100 records each) were returned until the API stopped giving data.
I looked up their terms for fair use and experimented with 5- and 10-second delays between each HTTP request but the responses are still unpredictable.
So, yeah, this probably is all based on how I use their API but between the rate limit documentation which – to my knowledge I'm not over-using – I can't figure out what the problem is.
### Recently created Meetups have a natural growth-burst in their first days
I've only had data for the last 4 days but usually the fastest-growing ones were less than a month old. That totally makes sense, the organizer is excited, promotes the Meetup etc.
To counter-point this, I'm looking forward to having more than a week's worth of data, then a month to compare.
### I need more data; and data costs
The value of this is directly related to how much data I can report on. It'll take at least a week for me to really start using it for real. I think a report on the fastest-growing Meetup in the last month will even be more valuable.
Speaking of data, I quickly exceeded 10,000 rows of the free hobby database on Heroku. I added a $9/mo. mini database on the third day.
Each day, I'm adding ~2,600 Member Count records plus ~2,600 Reporting Cube records to load the UI fast. I guess, that's 5,200 records per day in total or 156,000 records per month. Nothing crazy but just interesting.
Anyways, just a few thoughts I wanted to share!
Let me know what you think.
Alexis