Like On Facebook

Breaking News API In Python.

how you canaccess breaking news, headlines and search for articles from over 30,000news sources and blogs with the news API in Python. Coming up next. (really cool intro music) There's an incredible amount of things that you can do with a News API such as showing users top and breaking news headlines researchingpotential clients for risk or opportunities natural languageprocessing and machine learning the sky is the limit and what I want to do today is quickly get you started so that you can beginusing the news API in Python today. I'm also going to give you a heads up onsome very important caveats that aren't necessarily clear in the documentation.So make sure you watch till the end to avoid some very big pitfalls. Let's go.The first stop is the news API website. It's really pretty easy to navigate andthere's a lot of good stuff here but there's a lot that's not really relevantright away. So I'm going to give you a quick tour of the site and introduce youto the sections that will be most relevant to you right now and then whenyou've played around with the API for a while and wants to do some more advancedsearching you can come back and get some more information.

If you look across thetop you'll see several tabbed links Getting Started. This section includes example code an output of the three endpoints which are(1)top headlines, (2) everything and  (3) sources and these are the three types of requeststhat you can make. You'll notice that the output is in JSON format. Ultimately thiswill get transformed into a Python dictionary. Some of the print functionsin these code examples appear to have been written in Python 2. So watchcopying and pasting some of this stuff. The examples also use the Requestpackage. You can do this. However, the News API package is even simpler to use and less verbose. This is the package I would recommend using especially since thispackage will allow you to call the endpoints as specific methods instead ofhaving to program the URL header. Documentation. This section provides alot of information about how to use the API. However, the pages that are going to be most useful to you are (1) the clientlibrary page for Python and (2) the reference page for each endpoint. If youclick on the Python page you can see an example of exactly how to get started

.You first install the package with pip Then to use it, you import News APIclient to set up the NewsApi object that will handle the request. You thenhave three methods available to you that correspond to the three end points wejust looked at: Top Headlines, Everything, and Sources. You can see these parametershere. These are not all the optional parameters, this is just anexample. To get more information about the parameters and what values you canactually use you'll need to navigate to the endpoints pages. So let's do that now.The endpoint pages are going to give you a sample output but more importantlythey're going to explain what all of the request parameters are and examples ofvalues that you can use for these parameters. I would recommend gettingvery familiar with these parameters especially because it will allow you toscope your query to the most relevant records. In addition to the parametersyou can find out what the response object is. You'll see this when I runsome actual code, but the response object in Python is a dictionary that includesall of these components. The first level of the dictionary contains three objects:a string, an integer, and an array of articles which in python is a list ofdictionaries containing the items that you see here. News Sources. If you want torestrict your query to a specific news source, then you can. The string that youshould pass to the source parameter is indicated here under each news source.

Youcan also specify the country as well. Pricing. This tab tells you what you getwith different price plans. I'm using a developer account which is free. Youcan see that there's some differences here but let me just point out some ofthe big ones. First of all, paying accounts get thelatest news in real time, whereas the developer account has a 15-minute timedelay. Not a big deal for me. The developer account can accessarticles up to a month old, but you'll need to pay to access anything olderthan that. So keep that in mind. With a developer account you can submit up to500 requests per day. This number is substantially higher for paying accounts...it's in the hundreds of thousands. This may not seem like a big deal butthere is a catch... so hang on till the end of this video and I'll demonstrate theundocumented search limit caps and also give some advice on how to optimize yourquery results in order to work around some of the developer account searchlimits. Login.

Finally is the login screen. After you register for an API keywhich I'll show you in just a second you can login and see a few interestingbits of information. First, you can see your accounttype and your API key. And apparently, you can regenerate an API key if it getslost, stolen, or whatever. Second, there's a chart that shows your usage history. Thisis important if you're making a lot of requests in a single day, especially withthat 500 requests per day limit with a developer account. Finally there's a feedthat shows Twitter updates for this API. Apparently a package was recentlydeveloped for "R". So that's cool if you're using R. Okay, now that we'vedone a tour of the website, let's get started. The first thing you're gonnawant to do is register for a free api key by clicking on the blue "get api key"button on the website.


A very brief form is going to open up and after you fillit out, NewsApi.org will email you your api key which you can then copy andpaste into your code. Next, you're gonna need to install the package. So open upyour command prompt type: pip install newsapi-python. And, after it'sinstalled, let's start coding. Open up your IDE code editor, Jupiter notebook,whatever it is that you want to code in. I'm going to import the package I need bytyping "from newsapi import NewsApiClient" I've saved my API key in amodule called "key" so I'm going to import that as well. I'm going to show you howto do something with one of the date fields so I'm going to import the "datetime" module. Finally I'm going to do something in Pandas at the end of thisvideo so let's import Pandas as well. Next let's create a news API clientobject and I'm gonna use this to handle the query request.Type newsapi = NewsApiClient(api_key= my_api_key) As I said I've saved my API key in a variable. You can enter yours here as astring if you wish. Next let's create a search request.


There are three differentmethods that you can use to fetch data with a newsapi object. (1) get_top_headlines, (2) get_everything, and (3) get_sources. These are the endpoints that we saw onthe News API website. I'm going to go with "get_everything". Type in data = newsapi.get_everything( type in "q='Jupiter lab', language='en', page_size=20)One thing to be careful about is if you are looking at the News Api site forparameters. The parameters on the News API site are in camelCase. However theauthor of the Python package has translated the parameters into a Pythonic snake_case convention so just be cognizant of that naming difference oryou'll be banging your head against the wall trying to figure out what theproblem is. Okay let's examine the objects. If I print the type of this dataobject you can see that it's a dict. And since that is the case I can use thekeys() method to see what keys are available to use. You can see that thereare three objects inside and if you remember from the documentation welooked at on the website, the status returns a string, the total resultsreturns an integer, and the articles object is an array, which in Python is alist of dictionaries.

If I access the values of these first two keys you'llsee that I'm getting "ok" as status and then I'm getting 22 as total results.you'll also notice that the field names are still in camelCase. If I print the type() of the articles object you can see that it's a list. If I show the first item in the articles list you can see that it's a dictionarywith key value pairs. So in order to make this a little bit easier I'm going tosave a copy of the articles to a new list I'm calling "articles". Now I'm going to iterate through each article in the listof articles and print an index number and then the article title. All of theother items in the dictionary can be accessed in the same way. Let's lookat the first article and print every key-value pair in that article.Everything here is pretty straightforward.

You can see the source,author, title, description, URL, URL to image, publish date, content. Let me justtalk a second about a couple of these things. The URL is a direct link to thearticle. So if I were to copy and paste that into the browser then it would takeme directly to the article. The URL to image is also a direct link but you canadd that image to your content. For example, if I add that URL to a markuptag you can see the image pop right up. You could tie this to your HTML tag aswell if you're doing web development. You can convert the publish date into a Python daytime object by importing the datetime library, as Ihave, and convert it with this strptime() method. This will not only let youprint a more user-friendly date but then you can manipulate the date and getvarious parts of it to suit your needs. If you look at the content object,you'll notice that it gives you quite a bit of text but not all of it. Insteadyou get something that indicates an additional number of words. I'm notcertain because I have a developer account, but I believe that this is asmuch content as you get, which in this case is only about 275 characters.However, I can verify that the search results do include all text in theoriginal articles. Finally I thought I'd show you one more cool thing that youcan do with this object.


Since these articles are stored in a list ofdictionaries, this is easily imported into a pandas dataframe bytyping df = pd.DataFrame(articles) And when it's in a dataframe, the entire world of data manipulation in pandas and numpyis available to you. Plus it just looks really nice. Okay, I know this is a longvideo but this last part is something that you really need to see if you wantto avoid getting burned by the free developer account search limits. Here we go. As of the making of this video, if I wereto search for the phrase "iPhone X", I would get approximately 5,000 plusresults and that's just in the last 30 days. NewsApi will distribute theseresults into a number of pages, similar to if you were to do a Google search.You can adjust the number of results that appear on each page up to 100max. When I tried this, I thought "okay, no problem, I'll just iterate over the pagesand extend the list results with the list.extend() method... and that will giveme the complete set of results." However, I ran into a problem. The developer accountis limited to a max of 100 results PERIOD. So if you have 5,000 results,you're only able to see the first 100. I couldn't find this limit posted anywhereon the website.


It may be there but I couldn't find it.However when I tried to pull in page two of the results, the error message wasvery clear... if I want more than 100 results I need to upgrade to a payingaccount. However there's a few things that you can do to make your search morerelevant and therefore reduce the size of your results. Firstmake sure that you're more specific with keywords. Instead of searching for "iPhoneX" perhaps search for "iPhone X reviews" or "iPhone X reviews space grey". You can also limit the time scale. You may only want to see the articlespublished in the last week. If so, you can adjust those settings. Perhaps youonly care about articles published in a particular source or magazine, or acertain language... These are all things that you can do to increase therelevance of your search results and therefore reduce the total volume ofresults, making those developer caps less of a nuisance. And that my friendsis everything you need to get started using News API in Python. Please checkthe video description for all relevant links to the API website anddocumentation. How do you plan on using this news API?Please leave your thoughts in the comments below.


If you enjoy this video,it would give my ego a boost if you would let me know by smashing that like buttonand subscribing to my channel. Talk to you later!

Post a comment

0 Comments