Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

latitude and longitude field query is broken #77

Closed
YifeiLu-1 opened this issue Oct 6, 2020 · 6 comments
Closed

latitude and longitude field query is broken #77

YifeiLu-1 opened this issue Oct 6, 2020 · 6 comments

Comments

@YifeiLu-1
Copy link
Collaborator

YifeiLu-1 commented Oct 6, 2020 •
edited
Loading

Hi,
I noticed that I cannot query latitude and longitude field of the daft rental listings. It leads to the failure of mapping functions.

To reproduce the process

import pandas as pd
from daftlistings import Daft, RentType

daft = Daft()

daft.set_county("Dublin City")
daft.set_listing_type(RentType.APARTMENTS)
daft.set_min_price(1000)
daft.set_max_price(1250)

listings = daft.search(fetch_all=False)

properties = []
for listing in listings:
    properties.append(listing.as_dict())

df = pd.DataFrame(properties)
print(df)

Here is the printout of df

search_type agent_id       id    price price_change  ... transport_routes latitude longitude ber_code commercial_area_size
0       rental        2  2085334  1021.00         None  ...              N/A     None      None     None                  N/A        
1       rental           2085235  1200.00         None  ...              N/A     None      None     None                  N/A        
2       rental           2085151  1150.00         None  ...              N/A     None      None     None                  N/A        
3       rental           2084854  1200.00         None  ...              N/A     None      None     None                  N/A        
4       rental           2084844  1000.00         None  ...              N/A     None      None     None                  N/A        
5       rental           2084714   999.35         None  ...              N/A     None      None     None                  N/A        
6       rental           2084460  1100.00         None  ...              N/A     None      None     None                  N/A        
7       rental           2084382  1250.00         None  ...              N/A     None      None     None                  N/A        
8         None     None     None  1200.00         None  ...              N/A     None      None     None                  N/A        
9       rental     2946  2084033  1250.00         None  ...              N/A     None      None     None                  N/A        
10      rental           2083780  1250.00         None  ...              N/A     None      None     None                  N/A        
11      rental     8357  2083396  1200.00         None  ...              N/A     None      None     None                  N/A        
12      rental     5469  2083390  1223.00         None  ...              N/A     None      None     None                  N/A        
13      rental      776  2083307  1250.00         None  ...              N/A     None      None     None                  N/A        
14      rental           2081711  1250.00         None  ...              N/A     None      None     None                  N/A        
15      rental           2081156  1200.00         None  ...              N/A     None      None     None                  N/A        
16      rental           2081144  1200.00         None  ...              N/A     None      None     None                  N/A        
17      rental    10994  2080533  1100.00         None  ...              N/A     None      None     None                  N/A        
18      rental      362  2080500  1000.00         None  ...              N/A     None      None     None                  N/A        
19      rental           2079985  1100.00         None  ...              N/A     None      None     None                  N/A        

[20 rows x 31 columns]
@sdfordham
Copy link
Collaborator

The JS changed again, the regex pattern needs to be altered, the relevant part is the trackingParam dict and the 'lattitude' and 'longitude' keys. Actually reverting commit 57142e5 from #57 should fix this as far as I can see.

@sdfordham
Copy link
Collaborator

The JS changed again, the regex pattern needs to be altered, the relevant part is the trackingParam dict and the 'lattitude' and 'longitude' keys. Actually reverting commit 57142e5 from #57 should fix this as far as I can see.

Actually what I wrote is not true, that code you posted works correctly for me and returns latitude and longitude values, @YifeiLuDublin can you check again? Although I get a lot of log errors now:

ERROR:root:'NoneType' object has no attribute 'find'
ERROR:root:Error getting the Ber Code. Error message: 'NoneType' object has no attribute 'find'
ERROR:root:Error getting commercial_area_size. Error message: 'NoneType' object has no attribute 'text'
ERROR:root:Error getting price_change. Error message: 'NoneType' object has no attribute 'text'
ERROR:root:Error getting agent. Error message: 'NoneType' object has no attribute 'text'
ERROR:root:Error getting agent_url. Error message: 'NoneType' object has no attribute 'find_all'
ERROR:root:Error getting contact_number. Error message: 'NoneType' object has no attribute 'attrs'
ERROR:root:Error getting contact_number. Error message: 'NoneType' object has no attribute 'attrs'
ERROR:root:Error getting commercial_area_size. Error message: 'NoneType' object has no attribute 'find'
ERROR:root:Error getting commercial_area_size. Error message: 'NoneType' object has no attribute 'find'
ERROR:root:Error getting shortcode. Error message: 'NoneType' object has no attribute 'contents'
ERROR:root:Error getting date_insert_update. Error message: 'NoneType' object has no attribute 'contents'
ERROR:root:Error getting views. Error message: 'NoneType' object has no attribute 'contents'
ERROR:root:Error getting posted_since. Error message: 'NoneType' object has no attribute 'text'

@YifeiLu-1
Copy link
Collaborator Author

The JS changed again, the regex pattern needs to be altered, the relevant part is the trackingParam dict and the 'lattitude' and 'longitude' keys. Actually reverting commit 57142e5 from #57 should fix this as far as I can see.

Actually what I wrote is not true, that code you posted works correctly for me and returns latitude and longitude values, @YifeiLuDublin can you check again? Although I get a lot of log errors now:

ERROR:root:'NoneType' object has no attribute 'find'
ERROR:root:Error getting the Ber Code. Error message: 'NoneType' object has no attribute 'find'
ERROR:root:Error getting commercial_area_size. Error message: 'NoneType' object has no attribute 'text'
ERROR:root:Error getting price_change. Error message: 'NoneType' object has no attribute 'text'
ERROR:root:Error getting agent. Error message: 'NoneType' object has no attribute 'text'
ERROR:root:Error getting agent_url. Error message: 'NoneType' object has no attribute 'find_all'
ERROR:root:Error getting contact_number. Error message: 'NoneType' object has no attribute 'attrs'
ERROR:root:Error getting contact_number. Error message: 'NoneType' object has no attribute 'attrs'
ERROR:root:Error getting commercial_area_size. Error message: 'NoneType' object has no attribute 'find'
ERROR:root:Error getting commercial_area_size. Error message: 'NoneType' object has no attribute 'find'
ERROR:root:Error getting shortcode. Error message: 'NoneType' object has no attribute 'contents'
ERROR:root:Error getting date_insert_update. Error message: 'NoneType' object has no attribute 'contents'
ERROR:root:Error getting views. Error message: 'NoneType' object has no attribute 'contents'
ERROR:root:Error getting posted_since. Error message: 'NoneType' object has no attribute 'text'

I just checked and it still returns None for latitude and longitute

@YifeiLu-1
Copy link
Collaborator Author

The JS changed again, the regex pattern needs to be altered, the relevant part is the trackingParam dict and the 'lattitude' and 'longitude' keys. Actually reverting commit 57142e5 from #57 should fix this as far as I can see.

Actually what I wrote is not true, that code you posted works correctly for me and returns latitude and longitude values, @YifeiLuDublin can you check again? Although I get a lot of log errors now:

ERROR:root:'NoneType' object has no attribute 'find'
ERROR:root:Error getting the Ber Code. Error message: 'NoneType' object has no attribute 'find'
ERROR:root:Error getting commercial_area_size. Error message: 'NoneType' object has no attribute 'text'
ERROR:root:Error getting price_change. Error message: 'NoneType' object has no attribute 'text'
ERROR:root:Error getting agent. Error message: 'NoneType' object has no attribute 'text'
ERROR:root:Error getting agent_url. Error message: 'NoneType' object has no attribute 'find_all'
ERROR:root:Error getting contact_number. Error message: 'NoneType' object has no attribute 'attrs'
ERROR:root:Error getting contact_number. Error message: 'NoneType' object has no attribute 'attrs'
ERROR:root:Error getting commercial_area_size. Error message: 'NoneType' object has no attribute 'find'
ERROR:root:Error getting commercial_area_size. Error message: 'NoneType' object has no attribute 'find'
ERROR:root:Error getting shortcode. Error message: 'NoneType' object has no attribute 'contents'
ERROR:root:Error getting date_insert_update. Error message: 'NoneType' object has no attribute 'contents'
ERROR:root:Error getting views. Error message: 'NoneType' object has no attribute 'contents'
ERROR:root:Error getting posted_since. Error message: 'NoneType' object has no attribute 'text'

Is there any chance that you are still using an older version of the code? I tried the latest master branch and the pip install version. They both dont return the correct latitude and longitude.

@sdfordham
Copy link
Collaborator

I am on latest branch master.
E.g. if you look in source for https://www.daft.ie/dublin/apartments-for-rent/brittas/brittas-brittas-dublin-2089671/ which is first result from running your code, the relevant script is still there

<script>
    var maptiler = null;
    var loadMaptiler = function () {

        if (maptiler !== null) {
            return false;
        }

        maptiler = new mapboxgl.Map({
            container: 'MapContainer',
            style: 'https://api.maptiler.com/maps/streets/style.json?key=91rAQ4WIytC6QimSK1Ty',
            center: [-6.456031899999992, 53.2383846],
            zoom: 16
        });

        maptiler.on('load', function() {
            $('.MapActions').show();
            // add zoom controls
            var nav = new mapboxgl.NavigationControl();
            maptiler.addControl(nav, 'top-left');

            // add fullscreen control
            var fullScreen = new mapboxgl.FullscreenControl();
            maptiler.addControl(fullScreen, 'top-right');

            // add standard marker
            new mapboxgl.Marker()
                .setLngLat([-6.456031899999992, 53.2383846])
                .addTo(maptiler);

            maptiler.scrollZoom.disable();
        });

        maptiler.on('click', function() {
            if (this.scrollZoom.isEnabled() === false) {
                this.scrollZoom.enable();
            }
        });
        maptiler.on('mouseout', function() {
            if (this.scrollZoom.isEnabled() === true) {
                this.scrollZoom.disable();
            }
        })
    }

    window.addEventListener('load', loadMaptiler);
</script>

Matching on the .setLngLat method

daftlistings/daftlistings/listing.py

Line 437 in 729d0f4

if "setLngLat" in script.text:

So there is no reason it should fail as far as I can see.

@YifeiLu-1
Copy link
Collaborator Author

I am on latest branch master.
E.g. if you look in source for https://www.daft.ie/dublin/apartments-for-rent/brittas/brittas-brittas-dublin-2089671/ which is first result from running your code, the relevant script is still there

<script>
    var maptiler = null;
    var loadMaptiler = function () {

        if (maptiler !== null) {
            return false;
        }

        maptiler = new mapboxgl.Map({
            container: 'MapContainer',
            style: 'https://api.maptiler.com/maps/streets/style.json?key=91rAQ4WIytC6QimSK1Ty',
            center: [-6.456031899999992, 53.2383846],
            zoom: 16
        });

        maptiler.on('load', function() {
            $('.MapActions').show();
            // add zoom controls
            var nav = new mapboxgl.NavigationControl();
            maptiler.addControl(nav, 'top-left');

            // add fullscreen control
            var fullScreen = new mapboxgl.FullscreenControl();
            maptiler.addControl(fullScreen, 'top-right');

            // add standard marker
            new mapboxgl.Marker()
                .setLngLat([-6.456031899999992, 53.2383846])
                .addTo(maptiler);

            maptiler.scrollZoom.disable();
        });

        maptiler.on('click', function() {
            if (this.scrollZoom.isEnabled() === false) {
                this.scrollZoom.enable();
            }
        });
        maptiler.on('mouseout', function() {
            if (this.scrollZoom.isEnabled() === true) {
                this.scrollZoom.disable();
            }
        })
    }

    window.addEventListener('load', loadMaptiler);
</script>

Matching on the .setLngLat method

daftlistings/daftlistings/listing.py

Line 437 in 729d0f4

if "setLngLat" in script.text:

So there is no reason it should fail as far as I can see.

Somehow the map function isn't present in my PC but shows up on my phone. I guess my PC get banned for crawling too much?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sdfordham @YifeiLu-1