Connect with us

Real Estate Technology

Real estate data scraping back in a big way with new bots, surprising new tactics

Data scraping was a hot issue in the industry once upon a time, but the solutions fixed the problems. There are new bots with new tactics, and our industry has some vulnerabilities – here’s what we all need to do.



data scraping

Property sellers expect that, when they give an agent their listing information, it will be used to market and sell their property. While that means that agents need to give the listing information exposure on the Internet, it is the agent’s responsibility to take reasonable steps that the data is not ‘scraped’ off of their website and used for illicit purposes, such as direct-marketing the seller, display on unapproved places on the Internet, and other undesirable uses.

Realtors, this means ensuring that your website and software providers have taken steps to ensure that the data is not harvested by malicious software (“bots”). This is mandated by MLS rules for Virtual Office Websites (VOWs) but not yet for other displays. But now that IDX rules are changing to allow sold data, IDX should definitely be re-examined as well.

The scraping issue of yesterday vs. today

The ‘scraping’ issue was the center of attention in our industry a few years ago, when several MLSs went up against a nationwide data ‘scraper’ that foolishly re-posted the stolen data online where it could be found.

Still, it cost these MLSs over 10 million dollars and a lot of time spent in court to get this one scraper to stop. But most of the scrapers’ work product never sees the light of day, so we can’t easily find them and go after them. Proactively stopping the bots is the only way we can deal with this problem.

Where are the bots now a problem?

  • MLS systems – past the login, but also prospecting and client collaboration features, and the framed IDX solutions some vendors offer
  • MLS/Association consumer facing websites with listings (not to mention the member roster)
  • IDX sites
  • Virtual Office Websites (VOW)
  • Publishers / Portals

Stopping those bots is not easy for a developer or webmaster

Even just a few years ago, it was easier. A bot wouldn’t look (to the web server) like a real web browser. A bot would look at too many listings from one IP address, or look through them faster than any human ever could.

You can still catch a few of the less sophisticated bots by watching for those kinds of things – but most of the scrapers have moved beyond that level of sophistication, and it’s all too easy to block the good bots you want crawling your site, like search engines.

These days, the bots may be written to automate the activities of real web browsers, making it harder to distinguish bot traffic from people traffic. The bots may be deployed on thousands of computers with IP addresses that may belong to, or be re-deployed to, actual legitimate users – so blocking an IP address is no longer effective.

And, instead of looking at thousands of listings from one computer, the bots can now look at just a few listings from many computers – so old fashioned “rate limiting” and review of how many listings were viewed by one computer no longer help us differentiate between bots and real people.

How to protect your site from these bots

There are a variety of companies that specialize in stopping the scrapers’ bots. At one end, there’s Sentor, which is good and used by – but way too expensive for the smaller non-enterprise-level companies that make up our industry. Another is Distil Networks, which has a number of large scale platforms protected as well as MLSs, IDX sites, and brokerages and is highly accurate and effective.

This morning I was just reviewing data on an IDX vendor protected by Distil Networks. I saw that, over the past two months, over 7 million page requests had been made by malicious bots (differentiated from the good search engine bots) – over a million requests made by one bot alone!

And that’s just one IDX vendor that had taken the step to implement an anti-scraping solution. What is going to happen when that vendor’s protections are ratcheted up for blocking bots? I will tell you – the bad guys will move on to an easier target. Is your site an easy target for web scraping? We’re all in this together, and everybody needs to do their part.


Matt Cohen has been with Clareity Consulting for over 17 years, consulting for many of the real estate industry’s top Associations, MLSs, franchises, large brokerages and technology companies. Many clients look to Matt for help with system selection and negotiation. Technology providers look to Matt for assistance with product planning, software design, quality assurance, usability, and information security assessments. Matt has spoken at many industry events, has been published as an author in Stefan Swanepoel’s “Trends” report and many other publications, and has been honored by Inman News, being listed as one of the 100 Most Influential Real Estate Leaders.

Real Estate Technology

Camera that spins, zooms on its own – perfect for remote working

(TECHNOLOGY) Video conferencing tool, Meeting Owl Pro, helps bridge the communication gap in remote teams with a camera that spins and zooms on who is speaking



remote working tool

As someone who has spent the bulk of their career in a remote setting, I’ve definitely noticed some elements of communication that differ from an on-site job. Rather than being able to pop into your boss or coworker’s offices, you have to shoot them an email or try and catch them on their phone.

Enter the COVID-19 times…

Remote work can get even more complicated when the whole team is remote and you’re attempting to get everyone on the same page. Or, think about when you’re working remote and you have to video conference into a meeting with people who are on site. Even with video conference chatting, there can still be some kinks in trying to follow the voices and stay on top of who’s speaking.

Luckily, Owl Labs took this issue into consideration and whipped up Meeting Owl Pro and Meeting Owl First-Gen, which are the only 360° camera, mic, speaker smart devices on the market today. This is ideal for teams who have both an on-site and remote team collaborating together (or will again soon).

The Owl sits in the middle of the on-site table and moves in a complete 360° manner that picks up who is speaking and moves the camera to focus on them. That way, the people who are calling in remotely can clearly see who is speaking and follow the meeting as if they’re actually in the room.

Both versions of Meeting Owl are compatible with the big video conferencing platforms, such as: Zoom, Google, Skype, etc. They operate as plug and play devices that connect to Wi-Fi, which allows for automatic updates.

According to the tech team, the specifications of the Meeting Owl Pro 360° smart camera has an enhanced Owl Intelligence System™, 2X sharper camera with 1080p resolution and 2X louder 360° in-room speaker. With the Meeting Owl Pro, customers can expect their meeting spaces to become increasingly intelligent over time with new smart integrations and capabilities, the first of those features being a new Smart Zooming functionality that identifies, locates, and magnifies the person speaking.

This will help to bridge common communication barriers that are felt between on-site collaboration with remote teams. Additionally, it will help with team-building as everyone will have more of an opportunity to spend face-to-face time with their coworkers who work remote from them.

Continue Reading

Real Estate Technology

AI being used to better understand the COVID-19 pandemic

(REAL ESTATE TECHNOLOGY) The intersection of the coronavirus and AI is here and it’s trying to making the world a better place. AI can read and collect data faster than humans.



AI and the coronavirus

As the world scrambles to figure out the best way to prevent the spread of the Coronavirus and halt the ongoing pandemic, researchers are searching for a cure. In order to do so, it is beneficial to track patterns in the virus’s behavior.

Since the virus’s initial spread in December, thousands of papers have been written about the Coronavirus, or COVID-19. It would take weeks, if not months, for a person to collate all that information into some valuable data. This is where AI comes in to help.

Earlier this week the White House announced that they were working in collaboration with tech companies and members of academia to provide the huge amount of Coronavirus research to AI researchers and their algorithms.

The AI will be able to comb through all of the research we have to-date and search for patterns that could help those in the research field to find a cure and those in the medical field to better treat their patients. According to Wired, companies and institutions such as the National Library of Medicine, Microsoft Research, and the Allen Institute for AI (AI2) are currently working to gather and prepare the nearly 30,000 papers related to the virus so they can be processed by AI.

Hopefully, the AI’s algorithms are able to see something we cannot. It could find new connections that researchers have not been able to make and speed up the process.

The CEO of AI2, Oren Etzioni, is hopeful that this project will show people the brighter side of AI. “High tech, in general, has gotten a bad rap, but something like this crisis shows how AI can potentially do a world of good.”

There is no question that AI can help us find some answers as we face this pandemic. Whether or not this can all be arranged quickly enough to actually help mitigate the current crisis is the question. Some are wondering if U.S. resources would be put to better use by helping to address the shortage of Coronavirus test kits.

Experts are imploring people to think of the big picture. The issue of open access to medical and scientific research has been a topic of conversation among researchers for years. The pandemic has only highlighted the need. Now governments across the globe are calling on scientific publishers to open access for research on the Coronavirus so we can all work together in this time of global crisis.

Continue Reading

Real Estate Technology

How telecom, power companies are helping customers during COVID-19 outbreak

(REAL ESTATE TECHNOLOGY) Telecom and power companies have also tried to help their customers, while the Coronavirus makes its rounds, by extending service.



telecom helps customers

According to Fox News, Mark Cuban, owner of the Dallas Mavericks, promised to take care of the employees at the American Airline Center during the NBA shutdown. The employees would be paid as if the games were still taking place. Many of the other leagues have made the same promise. Unfortunately, not every employee works for a business with those types of provisions.

Thankfully, many organizations are responding to the COVID-19 pandemic with special accommodations to help residential and commercial customers deal with financial issues and staying home to prevent the spread of the virus.

Companies helping out customers

Comcast is offering 60 days of free basic internet service to new customers. It’s also not disconnecting or charging late fees. Current customers are also being provided with unlimited data. AT&T is offering the same types of things for its users, unlimited data and suspending terminations, as is T-Mobile.

Closer to home, the City of Austin has decided not to disconnect customers’ services for non-payment. Evictions are also halted. Austin Energy is reconnecting users who were recently disconnected for non-payment. The company is working with customers to help them get on track with their electric bills.

Customers without power do need to reach out to Austin Energy for a courtesy reconnect, because the bill cannot be restarted without speaking to the customer. The Public Utility Commission has asked all Texas power companies to work with customers during these uncertain times. TXU Energy has also waived late fees and is extending payment due dates.

Contact your utility providers

If your finances are being affected due to COVID-19, talk to your landlord and other providers now to work out a plan going forward. You’ll have less stress when you know you won’t lose your power, water or home. Companies are willing to work with families who have been impacted by the pandemic.

Continue Reading

Our Parnters

Get The Daily Intel
in your inbox

Subscribe and get news and EXCLUSIVE content to your email inbox!

Still Trending

Get The American Genius
in your inbox

subscribe and get news and exclusive content to your email inbox