I spent months doing this after getting laid off in April 2025, and eventually I tried to automate it. I'm a frontend developer — scraping wasn't my thing, but I figured: write a script, search Google for companies that are hiring, visit their career pages, extract openings. Simple pipeline.
It didn't work. And the reason it didn't work taught me something I haven't seen anyone talk about.
The obvious approach
The plan was straightforward. Use Google to find companies matching my preferences — small-to-mid tech teams, product companies, remote-friendly — and then crawl into their websites to check for open positions.
Step one: Google search with queries like frontend developer hiring remote Europe. Step two: visit each result URL. Step three: look for job listings.
I hit two walls immediately.
The first was expected: Google blocks headless browsers. My Playwright script got detected and shut down within minutes. Fine — I switched to Serper.dev, a Google Search API wrapper, for the query step and kept Playwright for actually visiting career pages. Problem solved.
The second wall was the real one.
Google's SEO makes company-first job discovery structurally impossible
Here's what happens when you search Google with any query that contains hiring-related keywords — jobs, hiring, careers, open positions, anything in that semantic neighborhood.
Roughly 90% of the results are job-board aggregators. Indeed. LinkedIn Jobs. Glassdoor. RemoteOK. Wellfound. The actual company websites — the ones I wanted — are buried or absent entirely.
This isn't a bug. This is SEO working exactly as designed.
Job boards are SEO machines. Their entire business model depends on ranking for hiring-related queries. They have massive domain authority, thousands of interlinked job pages, dedicated SEO teams, and years of backlink accumulation. A 50-person logistics company in Poland with three open positions doesn't stand a chance against that, no matter how I phrase the query.
I tried every variation I could think of. Adding company-type filters, industry keywords, location qualifiers, negative keywords. It didn't matter. The moment Google detects job-search intent in a query, the results page becomes a job-board directory. The algorithm isn't broken — it's serving exactly what most people want when they search for jobs. It's just useless for what I needed.
This is the part that took me a while to accept: you cannot find company websites through Google if your query looks like a job search. The intent signal overrides everything else.
The pattern that changed my approach
While I was figuring all this out, I should mention how I actually got my current job — because the pattern matters.
Someone in a Telegram job-search group shared a link to a LinkedIn post. Not a LinkedIn Job listing — a regular post, from someone at the company, mentioning they were hiring. Almost no engagement. Low views, few likes, minimal comments. I applied, got interviews, got hired.
A few months later, still passively looking, the same thing happened: a community tip led to another low-traffic LinkedIn post, another opening that hadn't surfaced on any job board. I made it to the final rounds before getting ghosted. Different outcome, but identical discovery pattern.
Then months went by with no interview invitations at all. It wasn't until about a year after getting hired that I looked back at those two experiences side by side and recognized what they had in common: low-visibility postings — the ones that don't surface on job boards, don't show up in Google, and don't get thousands of applicants — were exactly the openings where I had the best odds. Not because the roles were better, but because the applicant pool was thin.
Every auto-apply tool I could find — AIHawk, LazyApply, LoopCV — targets the same crowded job portals. They're optimizing for volume on the same listings that already have hundreds or thousands of applicants. Nobody was solving the discovery problem.
Separating company discovery from opportunity detection
The root cause of my Google failure was clear once I named it: I was trying to do two things in one query. Find companies and check if they're hiring. Google can handle the first one fine. It's the second signal — the hiring intent — that triggers the aggregator takeover.
The fix was to fully separate these into two operations with different intents.
Harvest mode generates search queries with no hiring keywords at all. Instead of frontend developer hiring Poland, it searches for things like logistics SaaS company Poland tech team or healthtech startup Tallinn engineering. These queries return what I actually wanted: real company websites. The scraper extracts company names and domains from the results and saves them to a database. No job-checking happens here.
Recheck mode takes the companies already in the database and visits their career pages directly — probing /careers, /jobs, /work-with-us if no career URL is saved yet. This is where Playwright earns its keep: navigating real websites, handling JavaScript-rendered pages, checking for actual job listing signals.
Discover mode is the original keyword-based approach, kept as a supplementary channel. It occasionally surfaces something the other modes miss, but it's no longer the foundation.
The key insight was that Harvest builds the list and Recheck checks the list. Together they replicate my original pipeline intent — find companies, then check if they're hiring — but by splitting the query intent, Google's SEO bias stops being a problem.
Keeping the AI bill under control
Discover mode — the original single-pipeline approach — still has a role as a supplementary channel. But because it searches Google with hiring keywords, the results are noisy: aggregator pages, generic boilerplate, 404s mixed in with real career pages. Sending all of that to an LLM for scoring would be expensive and wasteful.
So in Discover mode, before any AI call happens, each page goes through a two-stage rule-based filter. The first stage is loose: is this page even career-related? If it's a product page or a 404, it gets dropped. The second stage is strict: does the page have signals of active listings? Things like "Apply Now" buttons, salary ranges, responsibility lists, "What you'll do" sections. Generic boilerplate ("we hire great engineers") doesn't pass.
Only pages that survive both stages get sent to the AI for scoring and matching against my profile. This means as the company list grows from 50 to 500 to 5,000, the AI cost stays roughly flat — the pre-filter absorbs the scale, not the LLM.
What I learned
The interesting takeaway isn't about job searching specifically. It's about what happens when you try to automate something and discover the infrastructure you're relying on has a structural bias that makes your use case impossible.
Google's search results aren't neutral. They're shaped by decades of SEO investment from companies whose business model is to dominate exactly the queries you're typing. That's not a conspiracy — it's market dynamics. But it means that certain discovery patterns are effectively invisible to search engines, and the only way to reach them is to decompose your query into parts that don't trigger the bias.
There's another angle here that I think is underappreciated. AI is making it faster and cheaper for non-tech companies to adopt technology. A hospital, a logistics firm, a retail chain — businesses that never had an engineering culture are now realistic places for developers to work, because the barrier to building and maintaining software has dropped. That means job opportunities for developers are becoming more distributed across industries, not less. But these companies don't post on Hacker News or get featured on tech Twitter. Their openings sit on their own career pages, buried under Google's aggregator wall, seen by almost no one.
That's exactly the niche Vacansearch is built for. The tool that finds a 50-person logistics company in Poland hiring a frontend developer — a listing that no job board indexed and no search engine surfaced — is solving a problem that's only going to get bigger as more non-tech companies start building their own products.
I suspect the aggregator-dominance problem applies to a lot of domains beyond job search, too. Anytime you're looking for primary sources and the search results are dominated by aggregators — news, research, reviews, real estate — the same structural issue likely exists. And the same fix might apply: separate the entity discovery from the attribute check.
Vacansearch is still a local-only tool running on my machine. It's early. But the discovery engine works, the company list is growing, and I'm finding openings I never would have seen through any job board.
Sometimes the most useful thing you build starts with a script that fails for the right reasons.