mirror of
https://github.com/alanorth/cgspace-notes.git
synced 2024-06-29 01:23:47 +02:00
36 lines
1.6 KiB
Markdown
36 lines
1.6 KiB
Markdown
|
---
|
||
|
|
title: "July, 2022"
|
||
|
|
date: 2022-07-02T14:07:36+03:00
|
||
|
|
author: "Alan Orth"
|
||
|
|
categories: ["Notes"]
|
||
|
|
---
|
||
|
|
|
||
|
|
## 2022-07-02
|
||
|
|
|
||
|
|
- I learned how to use the Levenshtein functions in PostgreSQL
|
||
|
|
- The thing is that there is a limit of 255 characters for these functions in PostgreSQL so you need to truncate the strings before comparing
|
||
|
|
- Also, the trgm functions I've used before are case insensitive, but Levenshtein is not, so you need to make sure to lower case both strings first
|
||
|
|
|
||
|
|
<!--more-->
|
||
|
|
|
||
|
|
- A working query checking for duplicates in the recent AfricaRice items is:
|
||
|
|
|
||
|
|
```console
|
||
|
|
localhost/dspace= ☘ SELECT text_value FROM metadatavalue WHERE dspace_object_id IN (SELECT uuid FROM item) AND metadata_field_id=64 AND levenshtein_less_equal(LOWER('International Trade and Exotic Pests: The Risks for Biodiversity and African Economies'), LEFT(LOWER(text_value), 255), 3) <= 3;
|
||
|
|
text_value
|
||
|
|
────────────────────────────────────────────────────────────────────────────────────────
|
||
|
|
International trade and exotic pests: the risks for biodiversity and African economies
|
||
|
|
(1 row)
|
||
|
|
|
||
|
|
Time: 399.751 ms
|
||
|
|
```
|
||
|
|
|
||
|
|
- There is a great [blog post discussing Soundex with Levenshtein](https://www.crunchydata.com/blog/fuzzy-name-matching-in-postgresql) and creating indexes to make them faster
|
||
|
|
- I want to do some proper checks of accuracy and speed against my trigram method
|
||
|
|
|
||
|
|
## 2022-07-03
|
||
|
|
|
||
|
|
- Start a harvest on AReS
|
||
|
|
|
||
|
|
<!-- vim: set sw=2 ts=2: -->
|