0

I haven't read Slashdot and SoylentNews for the past year, and would like to catch up of what I missed.

Question

Is there a Linux program that can fetch/download the past 12 months of front pages stories of Slashdot and/or SoylentNews?

1 Answer 1

1

For Slashdot you can fetch the page for a particular day with a url in the format:

https://slashdot.org/?issue=20200110

Where in this example the ?issue= is followed by a date in the format YYYYMMDD and using this I was able to go back to the beginning of 2000.

You can further filter it to a topic, in this example to the topic technology, by using the format:

https://technology.slashdot.org/?issue=20200416

Possible topics are:

  • devices
  • build
  • entertainment
  • technology
  • opensource
  • science
  • yro

Knowing this it is perfectly possible to manually, in a browser or using wget, to view the summary pages for each day. Of course for a whole year this would take a bit of typing so I would suggest automating this with a python script. However, please respect the loading of the servers by taking some time, (and ideally running during a quiet time).

There are a number of ways to do this but I would suggest using requests-html or possibly scrapy but if you are using requests_html don't forget to have a time.sleep between page fetches.

Example using requests-html that should get this years headlines as a summary page with the links to the headlines:

#! usr/bin python3
import time
import datetime
from requests_html import HTMLSession
url_base = "https://technology.slashdot.org/?issue=%04d%02d%02d"
start_date = datetime.datetime(2020, 1, 1) # Start new years day
day = datetime.timedelta(days=1)

session = HTMLSession()
url_base = "https://technology.slashdot.org/?issue=%s"

outfile = open("headlines.html", "wb")
# Put a html header in
outfile.write(b"""<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> 
 <head> 
 <title>title</title>
 <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
 <meta name="description" content="" /> <meta name="keywords" content="" /> 
 <meta name="robots" content="index,follow" /> 
 <link rel="stylesheet" type="text/css" href="styles.css" />
 </head>  
 <body>""")
theday = start_date
def mkheader(aday):
    """ Header for a day """
    return b"<h1>%s</h1>\n" % aday.strftime('%D-%m-%d').encode('utf-8')

while theday < datetime.datetime.now():
    page = session.get(url_base % theday.strftime("%Y%m%d"))
    elements = page.html.find('h2.story')
    outfile.write(mkheader(theday))
    for e in elements:
        outfile.write(e.html.replace('href="//', 'href="https://').encode())
        outfile.write(b"\n")
    theday += day
    time.sleep(10) # Ten seconds delay might need to add a random element

outfile.write(b"</body>\n</html>")
outfile.close()
4
  • 1
    The script fails for me. I get ` File "s.py", line 25 for e in elements:` and ` File "s.py", line 26 outfile.write(e.raw_html)` Commented Jun 13, 2020 at 8:43
  • 1
    @SandraSchlichting: Sorry there was a missing ) on line 24. Commented Jun 13, 2020 at 11:40
  • 1
    Now I get: Traceback (most recent call last): File "s.py", line 30, in <module> page = session.get(url_base % theday.strftime("%Y%m%d")) NameError: name 'session' is not defined Commented Jun 13, 2020 at 13:01
  • 1
    Missing line added! Sorry, pasting error. Also fixing up links for each headline. Commented Jun 13, 2020 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.