Connect with us

Real Estate Technology

Real estate data scraping back in a big way with new bots, surprising new tactics

Data scraping was a hot issue in the industry once upon a time, but the solutions fixed the problems. There are new bots with new tactics, and our industry has some vulnerabilities – here’s what we all need to do.



data scraping

Property sellers expect that, when they give an agent their listing information, it will be used to market and sell their property. While that means that agents need to give the listing information exposure on the Internet, it is the agent’s responsibility to take reasonable steps that the data is not ‘scraped’ off of their website and used for illicit purposes, such as direct-marketing the seller, display on unapproved places on the Internet, and other undesirable uses.

Realtors, this means ensuring that your website and software providers have taken steps to ensure that the data is not harvested by malicious software (“bots”). This is mandated by MLS rules for Virtual Office Websites (VOWs) but not yet for other displays. But now that IDX rules are changing to allow sold data, IDX should definitely be re-examined as well.

The scraping issue of yesterday vs. today

The ‘scraping’ issue was the center of attention in our industry a few years ago, when several MLSs went up against a nationwide data ‘scraper’ that foolishly re-posted the stolen data online where it could be found.

Still, it cost these MLSs over 10 million dollars and a lot of time spent in court to get this one scraper to stop. But most of the scrapers’ work product never sees the light of day, so we can’t easily find them and go after them. Proactively stopping the bots is the only way we can deal with this problem.

Where are the bots now a problem?

  • MLS systems – past the login, but also prospecting and client collaboration features, and the framed IDX solutions some vendors offer
  • MLS/Association consumer facing websites with listings (not to mention the member roster)
  • IDX sites
  • Virtual Office Websites (VOW)
  • Publishers / Portals

Stopping those bots is not easy for a developer or webmaster

Even just a few years ago, it was easier. A bot wouldn’t look (to the web server) like a real web browser. A bot would look at too many listings from one IP address, or look through them faster than any human ever could.

You can still catch a few of the less sophisticated bots by watching for those kinds of things – but most of the scrapers have moved beyond that level of sophistication, and it’s all too easy to block the good bots you want crawling your site, like search engines.

These days, the bots may be written to automate the activities of real web browsers, making it harder to distinguish bot traffic from people traffic. The bots may be deployed on thousands of computers with IP addresses that may belong to, or be re-deployed to, actual legitimate users – so blocking an IP address is no longer effective.

And, instead of looking at thousands of listings from one computer, the bots can now look at just a few listings from many computers – so old fashioned “rate limiting” and review of how many listings were viewed by one computer no longer help us differentiate between bots and real people.

How to protect your site from these bots

There are a variety of companies that specialize in stopping the scrapers’ bots. At one end, there’s Sentor, which is good and used by – but way too expensive for the smaller non-enterprise-level companies that make up our industry. Another is Distil Networks, which has a number of large scale platforms protected as well as MLSs, IDX sites, and brokerages and is highly accurate and effective.

This morning I was just reviewing data on an IDX vendor protected by Distil Networks. I saw that, over the past two months, over 7 million page requests had been made by malicious bots (differentiated from the good search engine bots) – over a million requests made by one bot alone!

And that’s just one IDX vendor that had taken the step to implement an anti-scraping solution. What is going to happen when that vendor’s protections are ratcheted up for blocking bots? I will tell you – the bad guys will move on to an easier target. Is your site an easy target for web scraping? We’re all in this together, and everybody needs to do their part.


Matt Cohen has been with Clareity Consulting for over 17 years, consulting for many of the real estate industry’s top Associations, MLSs, franchises, large brokerages and technology companies. Many clients look to Matt for help with system selection and negotiation. Technology providers look to Matt for assistance with product planning, software design, quality assurance, usability, and information security assessments. Matt has spoken at many industry events, has been published as an author in Stefan Swanepoel’s “Trends” report and many other publications, and has been honored by Inman News, being listed as one of the 100 Most Influential Real Estate Leaders.

Real Estate Technology

Your office could benefit from a more open floor plan

(TECHNOLOGY NEWS) Science proves that open floor plans are more conducive to office productivity, but will it work for everyone?



open office

If you walk into a tech startup, nine times out of ten you’ll find an open seating/bull-pen style seating. Whereas traditional work environments are divided up into departments with individual offices and cubicles, open office floor plans put all employees in the same room. Studies have shown that cubicles don’t increase productivity. As a matter of fact, people are more productive when they are sitting close together, but can see each other.

Pros of openness

Some of the advantages of an open office floor plan are obvious. These kinds of offices are economical because you can fit more people and more desks in less space, and because it is more efficient to heat, cool, and light one large room than several small rooms.

Open office plans also facilitate communication between managers and their employees, and between departments.

Rather than taking the stairs or hiking down the hall to collaborate with another person, you can simply holler across the room.

Cons of openness

Unfortunately, all of that hollering can sometimes be pretty distracting. A University of Sydney study found that half of workers in open offices say that the most frustrating part of their workplace is the “lack of sound privacy.”

Open offices are not only noisy, but are also less secure, since everyone can overhear one another.

Employees may get peeved if they can’t concentrate because of all the noise around them, or can’t make a phone call without being overheard.

Dr. Who inspired solution

A startup called Framery Acoustics offers a solution.

They create soundproof phone booths and meeting pods designed to complement open office floor plans.

One of the founders, who previously worked in an open office, complained that his boss talked too loudly on his cellphone. His boss replied, “Well, get me a phone booth.” Thus, Framery Acoustics was born.

Simple solutions

Framery Acoustics is just one company that offers a product suited to appease open office dissenters. Framery Acoustics isn’t ready to give up on openness and neither should you. So, when it comes time to return to your office (if you haven’t already), look for ways to make your office more flexible. Whether it is by providing a quiet capsule for private meetings and phone calls or just having a designated section for meeting, the solution is out there.

Compromising allows you to reap the benefits of an open office plan, while still ensuring that you and your officemates have privacy and quiet when it is needed.

Continue Reading

Real Estate Technology

3D printed homes are now gaining traction outside of the US and China

(TECHNOLOGY) Other countries are now using 3d printing to build homes to underscore their infrastructure. This shows the viability of the technology!



3D printing

Recently, we reported that Lennar was using 3D printing to build homes in Austin. In 2014, the BBC reported that China was printing up to 10 homes a day at the low cost of $5000 per home. This trend is making strides in the real estate market, even though there’s still a long way to go. In a move that should give the industry confidence in 3D printing, Indonesia’s Public Works and Public Housing (PUPR) Ministry announced that they are using concrete 3D printing to build homes in rural areas. Eventually, plans are in the works to construct schools.

Using 3D printing to build an infrastructure

Indonesia is the world’s fourth most populous country. As with most countries, housing expenses are climbing in both urban and rural areas. According to Habitat for Humanity, 11.3% of the population lives below the poverty line. For comparison, in September, the U.S. Census Bureau released information that the U.S poverty rate increased to 11.4%, one percentage point over the same time in 2020. Affordable housing is a problem in Indonesia.

“This technology really helps us, so we can build faster, more accurately, and with precision,’ explains Kusumastuti, Indonesia’s Director General of Human Settlements.” The PUPR reports that 3D printing reduces waste and improves construction quality. Considering that up to 70% of housing is built by individuals, not private developers or the government, using 3D printing under the PUPR Ministry is an upgrade in a country that deals with many types of economic disasters, due to its climate.

3D printing’s potential for real estate

As 3D printing is used in more construction projects, not only in the U.S. and China, it’s hoped that the real estate industry embraces the technology. Indonesia isn’t the only country that is trying out 3D printing. 14Trees constructed a school in Malawi using this method already, with the project taking around 18 hours. The company is undertaking more projects in Africa using this technology and more companies are building houses using 3D printing in the United States. It will be exciting to watch how this plays out in the various markets.

Continue Reading

Real Estate Technology

Why everyone and their mother own spy machines (aka smart speakers)

(REAL ESTATE TECHNOLOGY) Regardless of privacy issues with them, what does information about smart speakers, ownership, and usage tell us about future trends?



smart speakers scare me

I don’t trust smart speakers, but even I can (begrudgingly) admit why they might be convenient. With just a simple wake word, I would be able to do anything from inquire about the weather or turn down my own music from across the room. And the thing is, plenty of people have bought into this sort of sales pitch. In fact, the worldwide revenue of smart speakers more than doubled between 2017 and 2018. And it’s projected that by 2022, the total revenue from smart speakers will reach almost $30 billion.

With over 25% of adults in the United States owning at least one smart speaker, it’s worth figuring out how we’re using this new tech…and how it could be used against us.

First things first: Despite the horror stories we hear about voice-command shopping – like when a pet parrot figured out how to make purchases on Alexa – people aren’t really using their smart speakers to buy things. In fact, in the list of top ten uses for a smart speaker, making a purchase is at the bottom.

Before you breathe a sigh of relief, though, it’s worth knowing where advertisements might crop up in more subtle places.

Sure, people aren’t using their smart speakers to make many purchases, but they’re still using the speakers for other things – primarily asking questions and getting updates on things like weather and traffic. And I get it, why scroll through the internet looking for an answer that Alexa might be able to pull up for you instantly?

That said, it also provides marketers with a great opportunity to advertise to you in a way that feels conversational. Imagine asking about a wait time for a popular restaurant. If the wait is too long, it creates the perfect opportunity for Alexa to suggest UberEats as an alternative (promotion paid for by UberEats, of course).

Don’t get me wrong, this is already happening when you search Google on your phone or computer. Search for a tire company, for instance, and the competitors are sure to appear in your results. But as more and more consumers start turning their attention to smart speakers, it’s worth being aware that they won’t be the only ones.

Continue Reading

Our Partners

Get The Daily Intel
in your inbox

Subscribe and get news and EXCLUSIVE content to your email inbox!

Still Trending

Get The American Genius
in your inbox

subscribe and get news and exclusive content to your email inbox