mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-10-23 01:13:01 +02:00
25 lines
1.3 KiB
Markdown
25 lines
1.3 KiB
Markdown
|
---
|
||
|
|
title: "October, 2024"
|
||
|
|
date: 2024-10-03T11:01:00+03:00
|
||
|
|
author: "Alan Orth"
|
||
|
|
categories: ["Notes"]
|
||
|
|
---
|
||
|
|
|
||
|
|
## 2024-10-03
|
||
|
|
|
||
|
|
- I had an idea to get abstracts from OpenAlex
|
||
|
|
- For [copyright reasons they don't include plain abstracts](https://docs.openalex.org/api-entities/works/work-object#abstract_inverted_index), but the [pyalex](https://github.com/J535D165/pyalex) library can convert them on the fly
|
||
|
|
|
||
|
|
<!--more-->
|
||
|
|
|
||
|
|
- I filtered for journal articles that were Creative Commons and missing abstracts:
|
||
|
|
|
||
|
|
```console
|
||
|
|
$ csvcut -c 'id,dc.title[en_US],dcterms.abstract[en_US],cg.identifier.doi[en_US],dcterms.type[en_US],dcterms.language[en_US],dcterms.license[en_US]' ~/Downloads/2024-09-30-cgspace.csv | csvgrep -c 'dcterms.type[en_US]' -r '^Journal Article$' | csvgrep -c 'cg.identifier.doi[en_US]' -r '^.+$' | csvgrep -c 'dcterms.license[en_US]' -r '^CC-' | csvgrep -c 'dcterms.abstract[en_US]' -r '^$' | csvgrep -c 'dcterms.language[en_US]' -r '^en$' | grep -v "||" | grep -v -- '-ND' | grep -v -E 'https://doi.org/10.(2499|4160|17528)/' > /tmp/missing-abstracts.csv
|
||
|
|
```
|
||
|
|
|
||
|
|
- Then wrote a script to get them from OpenAlex
|
||
|
|
- After inspecting and cleaning a few dozen up in OpenRefine (removing "Keywords:" and copyright, and HTML entities, etc) I managed to get about 440
|
||
|
|
|
||
|
|
<!-- vim: set sw=2 ts=2: -->
|