> ## Documentation Index
> Fetch the complete documentation index at: https://beta-docs-prod.indices.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Poll letting agents for new properties
> Build an automated system to monitor letting agents for new rental properties that match your criteria
Property hunting is time-consuming and competitive. This example shows how to build an automated system that continuously monitors letting agents' websites for new properties matching your specific criteria, giving you a competitive advantage in fast-moving rental markets.
## Use case overview
You're a property investor or tenant looking for 3-bedroom flats in Edinburgh with specific requirements. Rather than manually checking multiple letting agents multiple times a day, you'll create an automated scraper that:
* Monitors letting agents' websites for new properties
* Filters results by your criteria (bedrooms, location, price range)
* Extracts structured data including address, description, pricing, and availability
* Runs on a schedule to catch new listings as soon as they appear
## How it works
Define what data you want to extract and from which website.
Run the task with specific criteria to get current listings.
Set up automated execution to catch new properties immediately.
## Implementation
### Step 1: Create the scraping task
First, create a task that can extract property data from a letting agent's website. We'll use [Southside Management](https://southsidemanagement.com) as an example.
Let's define our input schema, so we can filter the results by bedrooms/location/price range, and the output schema to ensure a consistent data structure in the application code.
```json Input Schema expandable theme={null}
{
"type": "object",
"properties": {
"min_bedrooms": {
"type": "integer",
"description": "Minimum number of bedrooms"
},
"max_bedrooms": {
"type": "integer",
"description": "Maximum number of bedrooms"
},
"max_rent": {
"type": "number",
"description": "Maximum monthly rent in GBP"
}
},
"required": ["min_bedrooms", "max_bedrooms", "max_rent"]
}
```
```json Output Schema expandable theme={null}
{
"type": "array",
"items": {
"type": "object",
"properties": {
"property_id": {
"type": "string",
"description": "Unique identifier from the website"
},
"address": {
"type": "string",
"description": "Full property address"
},
"description": {
"type": "string",
"description": "Property description"
},
"bedrooms": {
"type": "integer",
"description": "Number of bedrooms"
},
"bathrooms": {
"type": "integer",
"description": "Number of bathrooms"
},
"rental_cost": {
"type": "number",
"description": "Monthly rental cost in GBP"
},
"cost_period": {
"type": "string",
"description": "Payment period (monthly, weekly, etc.)"
},
"available_from": {
"type": "string",
"description": "Date available to rent from"
},
"image_url": {
"type": "string",
"description": "URL to main property image"
},
"listing_url": {
"type": "string",
"description": "Link to full property details"
}
},
"required": [
"property_id", "address", "description", "bedrooms",
"bathrooms", "rental_cost", "cost_period", "available_from"
]
}
}
```
We can then create the task using the above schemas:
```bash cURL theme={null}
curl -X POST 'https://api.indices.io/v1beta/tasks' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"task": "Find available-to-rent flats in Edinburgh. Extract the property ID, address, description, number of bedrooms and bathrooms, rental cost with payment period, availability date, and main image URL if available. Focus only on properties from this letting agency - all information needed is available on their website.",
"website": "https://southsidemanagement.com/",
"input_schema": "{\"type\": \"object\", \"properties\": {\"min_bedrooms\": {\"type\": \"integer\", \"description\": \"Minimum number of bedrooms\"}, \"max_bedrooms\": {\"type\": \"integer\", \"description\": \"Maximum number of bedrooms\"}, \"max_rent\": {\"type\": \"number\", \"description\": \"Maximum monthly rent in GBP\"}}, \"required\": [\"min_bedrooms\", \"max_bedrooms\", \"max_rent\"]}",
"output_schema": "{\"type\": \"array\", \"items\": {\"type\": \"object\", \"properties\": {\"property_id\": {\"type\": \"string\", \"description\": \"Unique identifier from the website\"}, \"address\": {\"type\": \"string\", \"description\": \"Full property address\"}, \"description\": {\"type\": \"string\", \"description\": \"Property description\"}, \"bedrooms\": {\"type\": \"integer\", \"description\": \"Number of bedrooms\"}, \"bathrooms\": {\"type\": \"integer\", \"description\": \"Number of bathrooms\"}, \"rental_cost\": {\"type\": \"number\", \"description\": \"Monthly rental cost in GBP\"}, \"cost_period\": {\"type\": \"string\", \"description\": \"Payment period (monthly, weekly, etc.)\"}, \"available_from\": {\"type\": \"string\", \"description\": \"Date available to rent from\"}, \"image_url\": {\"type\": \"string\", \"description\": \"URL to main property image\"}, \"listing_url\": {\"type\": \"string\", \"description\": \"Link to full property details\"}}, \"required\": [\"property_id\", \"address\", \"description\", \"bedrooms\", \"bathrooms\", \"rental_cost\", \"cost_period\", \"available_from\"]}}}"
}'
```
```python Python lines theme={null}
import requests
import json
input_schema = {
"type": "object",
"properties": {
"min_bedrooms": {"type": "integer", "description": "Minimum number of bedrooms"},
"max_bedrooms": {"type": "integer", "description": "Maximum number of bedrooms"},
"max_rent": {"type": "number", "description": "Maximum monthly rent in GBP"}
},
"required": ["min_bedrooms", "max_bedrooms", "max_rent"]
}
output_schema = {
"type": "array",
"items": {
"type": "object",
"properties": {
"property_id": {"type": "string", "description": "Unique identifier from the website"},
"address": {"type": "string", "description": "Full property address"},
"description": {"type": "string", "description": "Property description"},
"bedrooms": {"type": "integer", "description": "Number of bedrooms"},
"bathrooms": {"type": "integer", "description": "Number of bathrooms"},
"rental_cost": {"type": "number", "description": "Monthly rental cost in GBP"},
"cost_period": {"type": "string", "description": "Payment period (monthly, weekly, etc.)"},
"available_from": {"type": "string", "description": "Date available to rent from"},
"image_url": {"type": "string", "description": "URL to main property image"},
"listing_url": {"type": "string", "description": "Link to full property details"}
},
"required": ["property_id", "address", "description", "bedrooms", "bathrooms", "rental_cost", "cost_period", "available_from"]
}
}
# Create the task
response = requests.post('https://api.indices.io/v1beta/tasks',
headers={
'Authorization': 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json'
},
json={
'task': 'Find available-to-rent flats in Edinburgh. Extract the property ID, address, description, number of bedrooms and bathrooms, rental cost with payment period, availability date, and main image URL if available. Focus only on properties from this letting agency - all information needed is available on their website.',
'website': 'https://southsidemanagement.com/',
'input_schema': json.dumps(input_schema),
'output_schema': json.dumps(output_schema)
}
)
task = response.json()
print(f"Task created: {task['id']}")
```
The API will immediately respond with an `id` and a `status` of `not_ready`.
```json Task creation response theme={null}
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"current_state": "not_ready",
"task": "Find available-to-rent flats in Edinburgh...",
"website": "https://southsidemanagement.com/",
"input_schema": "...",
"output_schema": "...",
"arguments": null,
"created_at": "2024-01-15T10:30:00Z",
"updated_at": "2024-01-15T10:30:00Z"
}
```
It typically takes 1-3 minutes for a task to be ready for usage, but may take up to 10 minutes.
### Step 2: Wait for task to be ready
Check the task status until it's ready for use:
```bash cURL theme={null}
curl -X GET 'https://api.indices.io/v1beta/tasks/550e8400-e29b-41d4-a716-446655440000' \
-H 'Authorization: Bearer YOUR_API_KEY'
```
```json Task Ready theme={null}
{
"id": "550e8400-e29b-41d4-a716-446655440000",
"current_state": "ready",
"task": "Find available-to-rent flats in Edinburgh...",
"website": "https://southsidemanagement.com/",
"input_schema": "...",
"output_schema": "...",
"arguments": {
"min_bedrooms": {"type": "integer"},
"max_bedrooms": {"type": "integer"},
"max_rent": {"type": "number"}
},
"created_at": "2024-01-15T10:30:00Z",
"updated_at": "2024-01-15T10:32:15Z"
}
```
You can programatically poll the `/tasks` endpoint to be notified when it's ready:
```python lines theme={null}
import requests
import asyncio
async def main():
task_id = ... # task_id from previous step
# poll the task until it's ready
while True:
response = requests.get(f"https://api.indices.io/v1beta/tasks/{task_id}")
task = response.json()
if task["current_state"] == "ready":
break
await asyncio.sleep(1)
```
### Step 3: Execute the task with your criteria
Now run the task with your filtering criteria:
```bash cURL theme={null}
curl -X POST 'https://api.indices.io/v1beta/runs' \
-H 'Authorization: Bearer YOUR_API_KEY' \
-H 'Content-Type: application/json' \
-d '{
"task_id": "550e8400-e29b-41d4-a716-446655440000",
"arguments": {
"min_bedrooms": 3,
"max_bedrooms": 3,
"max_rent": 2500
}
}'
```
```python Python theme={null}
import requests
import json
response = requests.post('https://api.indices.io/v1beta/runs',
headers={
'Authorization': 'Bearer YOUR_API_KEY',
'Content-Type': 'application/json'
},
json={
"task_id": "550e8400-e29b-41d4-a716-446655440000",
"arguments": {
"min_bedrooms": 3,
"max_bedrooms": 3,
"max_rent": 2500
}
}
)
```
```json Property Results theme={null}
{
"id": "run_123",
"task_id": "550e8400-e29b-41d4-a716-446655440000",
"arguments": {
"min_bedrooms": 3,
"max_bedrooms": 3,
"max_rent": 2500
},
"result_json": "[{\"property_id\": \"SM_2024_001\", \"address\": \"45 Marchmont Road, Edinburgh EH9 1HU\", \"description\": \"Spacious 3-bedroom flat in the heart of Marchmont with period features, high ceilings, and modern kitchen. Close to university and city centre.\", \"bedrooms\": 3, \"bathrooms\": 2, \"rental_cost\": 2200, \"cost_period\": \"monthly\", \"available_from\": \"15th February 2024\", \"image_url\": \"https://southsidemanagement.com/images/SM_2024_001_main.jpg\", \"listing_url\": \"https://southsidemanagement.com/properties/SM_2024_001\"}, {\"property_id\": \"SM_2024_002\", \"address\": \"12 Bruntsfield Place, Edinburgh EH10 4HN\", \"description\": \"Modern 3-bedroom apartment with en-suite master bedroom, open plan living, and private balcony. Excellent transport links.\", \"bedrooms\": 3, \"bathrooms\": 2, \"rental_cost\": 2400, \"cost_period\": \"monthly\", \"available_from\": \"1st March 2024\", \"image_url\": \"https://southsidemanagement.com/images/SM_2024_002_main.jpg\", \"listing_url\": \"https://southsidemanagement.com/properties/SM_2024_002\"}]",
"created_at": "2024-01-15T10:35:00Z",
"finished_at": "2024-01-15T10:35:12Z"
}
```
You can reuse the `task_id` to run the task many times. Different runs can use the same or different criteria.
## Setting up automated polling
To catch new properties as soon as they're listed, set up a scheduled system to run your task regularly:
```python Python theme={null}
import requests
import json
import time
from datetime import datetime
class PropertyPoller:
def __init__(self, api_key, task_id):
self.api_key = api_key
self.task_id = task_id
self.base_url = "https://api.indices.io"
self.headers = {
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json"
}
self.seen_properties = set()
def run_task(self, criteria):
"""Execute the scraping task with given criteria"""
response = requests.post(
f"{self.base_url}/runs",
headers=self.headers,
json={
"task_id": self.task_id,
"arguments": criteria
}
)
response.raise_for_status()
return response.json()
def check_for_new_properties(self, criteria):
"""Check for new properties and return only unseen ones"""
run_result = self.run_task(criteria)
properties = json.loads(run_result["result_json"])
new_properties = []
for prop in properties:
if prop["property_id"] not in self.seen_properties:
new_properties.append(prop)
self.seen_properties.add(prop["property_id"])
return new_properties
def poll_continuously(self, criteria, interval_minutes=30):
"""Poll for new properties at regular intervals"""
print(f"Starting property polling every {interval_minutes} minutes...")
while True:
try:
new_properties = self.check_for_new_properties(criteria)
if new_properties:
print(f"Found {len(new_properties)} new properties!")
for prop in new_properties:
self.notify_new_property(prop)
else:
print(f"No new properties found at {datetime.now()}")
time.sleep(interval_minutes * 60)
except Exception as e:
print(f"Error during polling: {e}")
time.sleep(60) # Wait 1 minute before retrying
def notify_new_property(self, property_data):
"""Handle new property notification (customize as needed)"""
print(f"NEW PROPERTY: {property_data['address']}")
print(f"Rent: £{property_data['rental_cost']}/{property_data['cost_period']}")
print(f"Available: {property_data['available_from']}")
print(f"Link: {property_data['listing_url']}")
print("-" * 50)
# Usage
poller = PropertyPoller(
api_key="your_api_key_here",
task_id="550e8400-e29b-41d4-a716-446655440000"
)
criteria = {
"min_bedrooms": 3,
"max_bedrooms": 3,
"max_rent": 2500
}
# Poll every 15 minutes
poller.poll_continuously(criteria, interval_minutes=15)
```
## Advanced configurations
### Multiple letting agents
Create separate tasks for different letting agents and poll them all:
```python theme={null}
agents = [
{
"name": "Southside Management",
"task_id": "550e8400-e29b-41d4-a716-446655440000"
},
{
"name": "DJ Alexander",
"task_id": "550e8400-e29b-41d4-a716-446655440001"
},
{
"name": "Citylets",
"task_id": "550e8400-e29b-41d4-a716-446655440002"
}
]
class MultiAgentPoller:
def __init__(self, api_key, agents):
self.api_key = api_key
self.agents = agents
self.pollers = {
agent["name"]: PropertyPoller(api_key, agent["task_id"])
for agent in agents
}
def poll_all_agents(self, criteria):
all_new_properties = []
for name, poller in self.pollers.items():
try:
new_props = poller.check_for_new_properties(criteria)
for prop in new_props:
prop["agent"] = name
all_new_properties.extend(new_props)
except Exception as e:
print(f"Error polling {name}: {e}")
return all_new_properties
```
### Smart notifications
Integrate with notification services for immediate alerts:
```python theme={null}
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
def send_email_alert(property_data, recipient_email):
"""Send email notification for new property"""
smtp_server = "smtp.gmail.com"
smtp_port = 587
sender_email = "your_email@gmail.com"
sender_password = "your_app_password"
message = MIMEMultipart()
message["From"] = sender_email
message["To"] = recipient_email
message["Subject"] = f"New Property Alert: {property_data['address']}"
body = f"""
New property matching your criteria:
Address: {property_data['address']}
Bedrooms: {property_data['bedrooms']}
Rent: £{property_data['rental_cost']}/{property_data['cost_period']}
Available: {property_data['available_from']}
Description: {property_data['description']}
View details: {property_data['listing_url']}
"""
message.attach(MIMEText(body, "plain"))
with smtplib.SMTP(smtp_server, smtp_port) as server:
server.starttls()
server.login(sender_email, sender_password)
server.send_message(message)
def send_slack_notification(property_data, webhook_url):
"""Send Slack notification for new property"""
import requests
payload = {
"text": f"🏠 New Property Alert",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": f"*{property_data['address']}*\n{property_data['bedrooms']} bed, {property_data['bathrooms']} bath\n£{property_data['rental_cost']}/{property_data['cost_period']}"
}
},
{
"type": "actions",
"elements": [
{
"type": "button",
"text": {"type": "plain_text", "text": "View Details"},
"url": property_data['listing_url']
}
]
}
]
}
requests.post(webhook_url, json=payload)
```
## Best practices
**Poll frequently** for competitive markets, but rate limit your requests to avoid being blocked.
Implement retry logic with exponential backoff. Log failures and continue polling other letting agents if one fails.
Store historical data to track market trends, price changes, and property availability patterns.
Start with broader criteria on API calls and shift filtering to your application logic. This reduces API calls while maintaining flexibility.
## Troubleshooting
**Cause**: Invalid JSON schema in input\_schema or output\_schema
**Solution**: Validate your JSON schemas using an online validator before creating the task. Ensure all required fields are properly defined.
**Cause**: The website structure is too complex or has anti-bot protections
**Solution**: Try simplifying your task description or contact support. Some websites may require special handling.
**Cause**: Your filtering criteria might be too restrictive, or the website has no matching properties
**Solution**: Test with broader criteria first, then gradually narrow down. Check the website manually to confirm properties exist.
**Cause**: The website displays data inconsistently across different property listings
**Solution**: Make your output schema more flexible by making fields optional where appropriate, and handle data cleaning in your application.
Start with a single letting agent and perfect your polling setup before expanding to multiple agents. This makes debugging much easier.
## Next steps
Once you have basic property polling working:
1. **Add data persistence** - Store results in a database to track price changes and availability
2. **Build a dashboard** - Create a web interface to view and manage your property alerts
3. **Implement ML filtering** - Use machine learning to score properties based on your preferences
4. **Scale to multiple cities** - Expand your monitoring to other locations and letting agents
5. **Add market analytics** - Track rental market trends and identify investment opportunities