Repository logo
 
Publication

Photo2Video: Semantic-Aware Deep Learning-Based Video Generation from Still Content

2022-03-10Journal article Open access
Simple item page
dc.contributor.authorViana, Paula
dc.contributor.authorAndrade, Maria Teresa
dc.contributor.authorCarvalho, Pedro
dc.contributor.authorVilaça, Luis
dc.contributor.authorTeixeira, Inês N.
dc.contributor.authorCosta, Tiago
dc.contributor.authorJonker, Pieter
dc.date.accessioned2023-01-19T12:23:44Z
dc.date.available2023-01-19T12:23:44Z
dc.date.issued2022-03-10
dc.description.abstractApplying machine learning (ML), and especially deep learning, to understand visual content is becoming common practice in many application areas. However, little attention has been given to its use within the multimedia creative domain. It is true that ML is already popular for content creation, but the progress achieved so far addresses essentially textual content or the identification and selection of specific types of content. A wealth of possibilities are yet to be explored by bringing the use of ML into the multimedia creative process, allowing the knowledge inferred by the former to influence automatically how new multimedia content is created. The work presented in this article provides contributions in three distinct ways towards this goal: firstly, it proposes a methodology to re-train popular neural network models in identifying new thematic concepts in static visual content and attaching meaningful annotations to the detected regions of interest; secondly, it presents varied visual digital effects and corresponding tools that can be automatically called upon to apply such effects in a previously analyzed photo; thirdly, it defines a complete automated creative workflow, from the acquisition of a photograph and corresponding contextual data, through the ML region-based annotation, to the automatic application of digital effects and generation of a semantically aware multimedia story driven by the previously derived situational and visual contextual data. Additionally, it presents a variant of this automated workflow by offering to the user the possibility of manipulating the automatic annotations in an assisted manner. The final aim is to transform a static digital photo into a short video clip, taking into account the information acquired. The final result strongly contrasts with current standard approaches of creating random movements, by implementing an intelligent content- and context-aware video.pt_PT
dc.description.sponsorshipThe work presented in this paper has been supported by the European Commission under contract number H2020-ICT-20-2017-1-RIA-780612 and by National Funds through the Portuguese funding agency, FCT—Fundação para a Ciência e a Tecnologia, within project LA/P/0063/2020.pt_PT
dc.description.versioninfo:eu-repo/semantics/publishedVersionpt_PT
dc.identifier.doi10.3390/jimaging8030068pt_PT
dc.identifier.urihttp://hdl.handle.net/10400.22/21682
dc.language.isoengpt_PT
dc.peerreviewedyespt_PT
dc.publisherMDPIpt_PT
dc.relationH2020-ICT-20-2017-1-RIA-780612pt_PT
dc.relationLA/P/0063/2020pt_PT
dc.relation.publisherversionhttps://www.mdpi.com/2313-433X/8/3/68pt_PT
dc.rights.urihttp://creativecommons.org/licenses/by-nc-nd/4.0/pt_PT
dc.subjectDeep Learningpt_PT
dc.subjectStorytellingpt_PT
dc.subjectAutomated content creationpt_PT
dc.subjectSemantic awarenesspt_PT
dc.subjectContext awarenesspt_PT
dc.subjectRoIpt_PT
dc.titlePhoto2Video: Semantic-Aware Deep Learning-Based Video Generation from Still Contentpt_PT
dc.typejournal article
dspace.entity.typePublication
oaire.citation.issue3pt_PT
oaire.citation.startPage68pt_PT
oaire.citation.titleJournal of Imagingpt_PT
oaire.citation.volume8pt_PT
person.familyNameViana
person.familyNamemagalhães da silva pinto de andrade
person.givenNamePaula
person.givenNamemaria teresa
person.identifier936138
person.identifier.ciencia-idEA17-B097-BD2E
person.identifier.ciencia-idBE1F-79C9-98F4
person.identifier.orcid0000-0001-8447-2360
person.identifier.orcid0000-0002-1363-5027
person.identifier.scopus-author-id7003678537
rcaap.rightsopenAccesspt_PT
rcaap.typearticlept_PT
relation.isAuthorOfPublication17ac1586-7589-4027-a541-3aea351fd6ae
relation.isAuthorOfPublication6defa1a4-c190-478b-ad8b-57d1e71a0f88
relation.isAuthorOfPublication.latestForDiscovery17ac1586-7589-4027-a541-3aea351fd6ae

Files

Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
ART_DEE_PMV_2022.pdf
Size:
6.86 MB
Format:
Adobe Portable Document Format
Download
License bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
license.txt
Size:
1.71 KB
Format:
Item-specific license agreed upon to submission
Description:
Download

Collections

ISEP - DEE - Artigos