feat(description): don't truncate by default (#668)

packages/metascraper-description/README.md

@@ -23,7 +23,7 @@ $ npm install metascraper-description --save
 ##### truncateLength
 
 Type: `number`<br>
-Default: `300`
+Default: `Number.MAX_SAFE_INTEGER`
 
 Truncate the value extracted to a maximum size.
 

packages/metascraper-helpers/index.js

@@ -115,8 +115,6 @@ const REGEX_LOCATION = /^[A-Z\s]+\s+[-—–]\s+/
 
 const REGEX_TITLE_SEPARATOR = /^[^|\-/•—]+/
 
-const TRUNCATE_MAX_LENGTH = 300
-
 const AUTHOR_MAX_LENGTH = 128
 
 const removeLocation = value => replace(value, REGEX_LOCATION, '')
@@ -253,7 +251,7 @@ const description = (value, opts) =>
 
 const getDescription = (
   str,
-  { truncateLength = TRUNCATE_MAX_LENGTH, ellipsis = '…', ...opts } = {}
+  { truncateLength = Number.MAX_SAFE_INTEGER, ellipsis = '…', ...opts } = {}
 ) => {
   let truncated = str.slice(0, truncateLength)
   if (truncated.length < str.length) truncated = truncated.trim() + ellipsis

packages/metascraper-helpers/test/index.js

@@ -58,11 +58,32 @@ test('.normalizeUrl', t => {
   )
   t.is(normalizeUrl('https://example.com/'), 'https://example.com/')
   t.is(normalizeUrl('https://example.com'), 'https://example.com/')
-  t.is(normalizeUrl('https://www.example.com', 'https://www.example.com/foo'), 'https://www.example.com/foo')
-  t.is(normalizeUrl('https://www.example.com', '/foo'), 'https://www.example.com/foo')
-  t.is(normalizeUrl('https://www.example.com', 'file.html'), 'https://www.example.com/file.html')
-  t.is(normalizeUrl('https://www.example.com', 'data:text/html;base64,PGh0bWw+SGVsbG8sIHdvcmxkITwvaHRtbD4='), 'data:text/html;base64,PGh0bWw+SGVsbG8sIHdvcmxkITwvaHRtbD4=')
-  t.is(normalizeUrl('https://www.example.com', 'javascript:alert(\'Hello, world!\');'), undefined)
+  t.is(
+    normalizeUrl('https://www.example.com', 'https://www.example.com/foo'),
+    'https://www.example.com/foo'
+  )
+  t.is(
+    normalizeUrl('https://www.example.com', '/foo'),
+    'https://www.example.com/foo'
+  )
+  t.is(
+    normalizeUrl('https://www.example.com', 'file.html'),
+    'https://www.example.com/file.html'
+  )
+  t.is(
+    normalizeUrl(
+      'https://www.example.com',
+      'data:text/html;base64,PGh0bWw+SGVsbG8sIHdvcmxkITwvaHRtbD4='
+    ),
+    'data:text/html;base64,PGh0bWw+SGVsbG8sIHdvcmxkITwvaHRtbD4='
+  )
+  t.is(
+    normalizeUrl(
+      'https://www.example.com',
+      "javascript:alert('Hello, world!');"
+    ),
+    undefined
+  )
   t.is(normalizeUrl('https://www.example.com', 'javascript:void(0)'), undefined)
 })
 
@@ -277,10 +298,12 @@ test('.isImageExtension', t => {
 test('.description', t => {
   t.is(
     description(
-      'Let me tell you the story of two investors, neither of whom knew each other, but whose paths crossed in an interesting way. Grace Groner was orphaned at age 12. She never married. She never had kids. She never drove a car. She lived most of her life alone in a one-bedroom house and worked her whole career as a secretary. She was, by all accounts, a lovely lady. But she lived a humble and quiet life. That made the $7 million she left to charity after her death in 2010 at age 100 all the more confusing. People who knew her asked: Where did Grace get all that money? But there was no secret. There was no inheritance. Grace took humble savings from a meager salary and enjoyed eighty years of hands-off compounding in the stock market. That was it. Weeks after Grace died, an unrelated investing story hit the news. Richard Fuscone, former vice chairman of Merrill Lynch’s Latin America division, declared personal bankruptcy, fighting off foreclosure on two homes, one of which was nearly 20,000 square feet and had a $66,000 a month mortgage. Fuscone was the opposite of Grace Groner; educated at Harvard and University of Chicago, he became so successful in the investment industry that he retired in his 40s to “pursue personal and charitable interests.” But heavy borrowing and illiquid investments did him in. The same year Grace Goner left a veritable fortune to charity, Richard stood before a bankruptcy judge and declared: “I have been devastated by the financial crisis … The only source of liquidity is whatever my wife is able to sell in terms of personal furnishings.” The purpose of these stories is not to say you should be like Grace and avoid being like Richard. It’s to point out that there is no other field where these stories are even possible. In what other field does someone with no education, no relevant experience, no resources, and no connections vastly outperform someone with the best education, the most relevant experiences, the best resources and the best connections? There will never be a story of a Grace Groner performing heart surgery better than a Harvard-trained cardiologist. Or building a faster chip than Apple’s engineers. Unthinkable. But these stories happen in investing. That’s because investing is not the study of finance. It’s the study of how people behave with money. And behavior is hard to teach, even to really smart people. You can’t sum up behavior with formulas to memorize or spreadsheet models to follow. Behavior is inborn, varies by person, is hard to measure, changes over time, and people are prone to deny its existence, especially when describing themselves. Grace and Richard show that managing money isn’t necessarily about what you know; it’s how you behave. But that’s not how finance is typically taught or discussed. The finance industry talks too much about what to do, and not enough about what happens in your head when you try to do it. This report describes 20 flaws, biases, and causes of bad behavior I’ve seen pop up often when people deal with money. 1. Earned success and deserved failure fallacy: A tendency to underestimate the role of luck and risk, and a failure to recognize that luck and risk are different sides of the same coin. I like to ask people, “What do you want to know about investing that we can’t know?” It’s not a practical question. So few people ask it. But it forces anyone you ask to think about what they intuitively think is true but don’t spend much time trying to answer because it’s futile. Years ago I asked economist Robert Shiller the question. He answered, “The exact role of luck in successful outcomes.” I love that, because no one thinks luck doesn’t play a role in financial success. But since it’s hard to quantify luck, and rude to suggest people’s success is owed to luck, the default stance is often to implicitly ignore luck as a factor. If I say, “There are a billion investors in the world. By sheer chance, would you expect 100 of them to become billionaires predominately off luck?” You would reply, “Of course.” But then if I ask you to name those investors – to their face – you will back down. That’s the problem. The same goes for failure. Did failed businesses not try hard enough? Were bad investments not thought through well enough?'
+      'Let me tell you the story of two investors, neither of whom knew each other, but whose paths crossed in an interesting way. Grace Groner was orphaned at age 12. She never married. She never had kids. She never drove a car. She lived most of her life alone in a one-bedroom house and worked her whole career as a secretary. She was, by all accounts, a lovely lady. But she lived a humble and quiet life. That made the $7 million she left to charity after her death in 2010 at age 100 all the more confusing. People who knew her asked: Where did Grace get all that money? But there was no secret. There was no inheritance. Grace took humble savings from a meager salary and enjoyed eighty years of hands-off compounding in the stock market. That was it. Weeks after Grace died, an unrelated investing story hit the news. Richard Fuscone, former vice chairman of Merrill Lynch’s Latin America division, declared personal bankruptcy, fighting off foreclosure on two homes, one of which was nearly 20,000 square feet and had a $66,000 a month mortgage. Fuscone was the opposite of Grace Groner; educated at Harvard and University of Chicago, he became so successful in the investment industry that he retired in his 40s to “pursue personal and charitable interests.” But heavy borrowing and illiquid investments did him in. The same year Grace Goner left a veritable fortune to charity, Richard stood before a bankruptcy judge and declared: “I have been devastated by the financial crisis … The only source of liquidity is whatever my wife is able to sell in terms of personal furnishings.” The purpose of these stories is not to say you should be like Grace and avoid being like Richard. It’s to point out that there is no other field where these stories are even possible. In what other field does someone with no education, no relevant experience, no resources, and no connections vastly outperform someone with the best education, the most relevant experiences, the best resources and the best connections? There will never be a story of a Grace Groner performing heart surgery better than a Harvard-trained cardiologist. Or building a faster chip than Apple’s engineers. Unthinkable. But these stories happen in investing. That’s because investing is not the study of finance. It’s the study of how people behave with money. And behavior is hard to teach, even to really smart people. You can’t sum up behavior with formulas to memorize or spreadsheet models to follow. Behavior is inborn, varies by person, is hard to measure, changes over time, and people are prone to deny its existence, especially when describing themselves. Grace and Richard show that managing money isn’t necessarily about what you know; it’s how you behave. But that’s not how finance is typically taught or discussed. The finance industry talks too much about what to do, and not enough about what happens in your head when you try to do it. This report describes 20 flaws, biases, and causes of bad behavior I’ve seen pop up often when people deal with money. 1. Earned success and deserved failure fallacy: A tendency to underestimate the role of luck and risk, and a failure to recognize that luck and risk are different sides of the same coin. I like to ask people, “What do you want to know about investing that we can’t know?” It’s not a practical question. So few people ask it. But it forces anyone you ask to think about what they intuitively think is true but don’t spend much time trying to answer because it’s futile. Years ago I asked economist Robert Shiller the question. He answered, “The exact role of luck in successful outcomes.” I love that, because no one thinks luck doesn’t play a role in financial success. But since it’s hard to quantify luck, and rude to suggest people’s success is owed to luck, the default stance is often to implicitly ignore luck as a factor. If I say, “There are a billion investors in the world. By sheer chance, would you expect 100 of them to become billionaires predominately off luck?” You would reply, “Of course.” But then if I ask you to name those investors – to their face – you will back down. That’s the problem. The same goes for failure. Did failed businesses not try hard enough? Were bad investments not thought through well enough?',
+      { truncateLength: 300 }
     ),
     'Let me tell you the story of two investors, neither of whom knew each other, but whose paths crossed in an interesting way. Grace Groner was orphaned at age 12. She never married. She never had kids. She never drove a car. She lived most of her life alone in a one-bedroom house and worked her whole…'
   )
+
   t.is(
     description(
       'He talks about the ocean and his favorite animal the cowrie...'

packages/metascraper-instagram/test/snapshots/index.js.md

@@ -28,7 +28,8 @@ Generated by [AVA](https://avajs.dev).
     {
       author: 'Willyrex',
       date: '2021-05-08T18:57:43.000Z',
-      description: '¿Listos? Nuevo #SORTEO en mi perfil 📱 Una vez más os traigo lo nuevo de OPPO, os presento el nuevo #OPPOFindX3Neo Su cámara es una auténtica locura, viene con Sensor IMX766 de 50 MP. Por la batería no os preocupéis, tiene carga SuperVOOC 2.0, 100% en 38 min. Además, tiene un diseño fino y ultrali…',
+      description: `¿Listos? Nuevo #SORTEO en mi perfil 📱 Una vez más os traigo lo nuevo de OPPO, os presento el nuevo #OPPOFindX3Neo Su cámara es una auténtica locura, viene con Sensor IMX766 de 50 MP. Por la batería no os preocupéis, tiene carga SuperVOOC 2.0, 100% en 38 min. Además, tiene un diseño fino y ultraligero 🏸 ¿Queréis uno como el mío? Participa ahora mismo: - Dale like a la publicación␊
+      - Sigue a OPPO @oppomobilees ¡Mucha suerte amig@s!`,
       image: 'https://instagram.fpit1-1.fna.fbcdn.net/v/t51.2885-15/e35/p1080x1080/183037094_508540027000559_4927547535791161246_n.jpg?tp=1&_nc_ht=instagram.fpit1-1.fna.fbcdn.net&_nc_cat=111&_nc_ohc=3GrGLwIF7PoAX-qfrOc&edm=AABBvjUBAAAA&ccb=7-4&oh=8a1021ad544a2f1a4bf911b4c371af85&oe=60BC6710&_nc_sid=83d603',
       lang: 'fr',
       logo: 'https://www.instagram.com/static/images/ico/favicon-192.png/68d99ba29cc8.png',

packages/metascraper-spotify/test/description.js

@@ -32,7 +32,7 @@ test('episode', async t => {
 
   t.is(
     metadata.description,
-    'In this Hasty Treat, Scott and Wes talk about modules in Node — what are they, how they’re different from browser modules, and more! Sentry - Sponsor If you want to know what’s happening with your errors, track them with Sentry. Sentry is open-source error tracking that helps developers monitor and…'
+    'In this Hasty Treat, Scott and Wes talk about modules in Node — what are they, how they’re different from browser modules, and more! Sentry - Sponsor If you want to know what’s happening with your errors, track them with Sentry. Sentry is open-source error tracking that helps developers monitor and fix crashes in real time. Cut your time on error resolution from five hours to five minutes. It works with any language and integrates with dozens of other services. Syntax listeners can get two months for free by visiting Sentry.io and using the coupon code “tastytreat”. Show Notes 3:06 - How were they done before? 5:11 - How do they work? 7:07 - How to use Modules in Node 9:57 - Gotchas 13:18 - What should you use? Links Node Node Modules Babel ESM Meteor Keystone MJS Tweet us your tasty treats! Scott’s Instagram LevelUpTutorials Instagram Wes’ Instagram Wes’ Twitter Wes’ Facebook Scott’s Twitter Make sure to include @SyntaxFM in your tweets'
   )
 })
 

packages/metascraper-spotify/test/snapshots/index.js.md

@@ -316,7 +316,7 @@ Generated by [AVA](https://avajs.dev).
       audio: 'https://podz-content.spotifycdn.com/audio/clips/4dcJc7WFMTmEWo5LmivA54/clip_546000_597200.mp3',
       author: 'Syntax - Tasty Web Development Treats',
       date: '2020-01-06T14:00:00.000Z',
-      description: 'In this Hasty Treat, Scott and Wes talk about modules in Node — what are they, how they’re different from browser modules, and more! Sentry - Sponsor If you want to know what’s happening with your errors, track them with Sentry. Sentry is open-source error tracking that helps developers monitor and…',
+      description: 'In this Hasty Treat, Scott and Wes talk about modules in Node — what are they, how they’re different from browser modules, and more! Sentry - Sponsor If you want to know what’s happening with your errors, track them with Sentry. Sentry is open-source error tracking that helps developers monitor and fix crashes in real time. Cut your time on error resolution from five hours to five minutes. It works with any language and integrates with dozens of other services. Syntax listeners can get two months for free by visiting Sentry.io and using the coupon code “tastytreat”. Show Notes 3:06 - How were they done before? 5:11 - How do they work? 7:07 - How to use Modules in Node 9:57 - Gotchas 13:18 - What should you use? Links Node Node Modules Babel ESM Meteor Keystone MJS Tweet us your tasty treats! Scott’s Instagram LevelUpTutorials Instagram Wes’ Instagram Wes’ Twitter Wes’ Facebook Scott’s Twitter Make sure to include @SyntaxFM in your tweets',
       image: 'https://i.scdn.co/image/ab6765630000ba8ab84c022aff150c336f73b395',
       lang: 'en',
       publisher: 'Spotify',

packages/metascraper/test/integration/anandtech/snapshots/index.js.md

@@ -12,7 +12,7 @@ Generated by [AVA](https://avajs.dev).
       audio: null,
       author: 'Anton Shilov',
       date: '2017-06-29T00:00:00.000Z',
-      description: 'This morning AMD is introducing their Ryzen PRO processors for business and commercial desktop PCs. The new lineup of CPUs includes the Ryzen 3 PRO, Ryzen 5 PRO and Ryzen 7 PRO families with four, six, or eight cores running at various frequencies. A superset to the standard Ryzen chips, the PRO chi…',
+      description: 'This morning AMD is introducing their Ryzen PRO processors for business and commercial desktop PCs. The new lineup of CPUs includes the Ryzen 3 PRO, Ryzen 5 PRO and Ryzen 7 PRO families with four, six, or eight cores running at various frequencies. A superset to the standard Ryzen chips, the PRO chips have the same feature set as other Ryzen devices, but also offer enhanced security, 24 months availability, a longer warranty and promise to feature better chip quality.',
       image: 'http://images.anandtech.com/doci/11591/amd_ryzen_pro_678_678x452.jpg',
       lang: null,
       logo: 'http://www.anandtech.com/content/images/rss_logo.png',

packages/metascraper/test/integration/bukvy/snapshots/index.js.md

@@ -12,7 +12,7 @@ Generated by [AVA](https://avajs.dev).
       audio: null,
       author: 'Буквы',
       date: '2019-01-06T07:43:59.000Z',
-      description: 'Президент Петр Порошенко заявил, что государство будет уважать религиозный выбор и вероисповедание каждого гражданина страны. Об этом он заявил в обращении после торжественной церемонии вручения в Стамбуле Томоса об автокефалии Православной церкви Украины. “Я как Президент гарантирую от имени госуда…',
+      description: 'Президент Петр Порошенко заявил, что государство будет уважать религиозный выбор и вероисповедание каждого гражданина страны. Об этом он заявил в обращении после торжественной церемонии вручения в Стамбуле Томоса об автокефалии Православной церкви Украины. “Я как Президент гарантирую от имени государства, что Украина будет уважать религиозный выбор и свободу вероисповедания каждого гражданина”, – подчеркнул он. Петр […]',
       image: 'https://bykvu.com/ru/kd_image_generate/141935.jpg',
       lang: null,
       logo: 'https://bykvu.com/wp-content/themes/bykvu/img/android-icon-192x192.png',

packages/metascraper/test/integration/health/snapshots/index.js.md

@@ -12,7 +12,7 @@ Generated by [AVA](https://avajs.dev).
       audio: null,
       author: null,
       date: '2011-04-07T12:00:00.000Z',
-      description: 'In December 2009 Cabinet agreed to strengthen the health sector’s focus on quality and safety by creating a Health Quality and Safety Commission. This regulatory impact statement considers what additional functions, powers, and funding mechanisms the Commission might need to achieve sustained qualit…',
+      description: 'In December 2009 Cabinet agreed to strengthen the health sector’s focus on quality and safety by creating a Health Quality and Safety Commission. This regulatory impact statement considers what additional functions, powers, and funding mechanisms the Commission might need to achieve sustained quality and safety improvement.',
       image: 'https://www.health.govt.nz/sites/all/themes/mohpub_bootstrap/images/mohlogo.svg',
       lang: 'en',
       logo: 'https://www.health.govt.nz/sites/all/themes/mohpub_bootstrap/favicon.ico',