Authors: John Janiczek, Dading Chong, Dongyang Dai, Arlo Faria1, Chao Wang, Tao
Wang, Yuzong Liu
Affiliation: Zoom Video Communications
Emails: john.janiczek@zoom.us, yuzong.liu@zoom.us
Abstract:
A text-to-speech (TTS) model trained to reconstruct speech given text tends towards predictions that are
close to
the average characteristics of a dataset, failing to model the variations that make human speech sound
natural.
This problem is magnified for zero-shot voice cloning, a task that requires training data with high variance
in
speaking styles. We build off of recent works which have used Generative Adversarial Networks (GAN) by
proposing a
Transformer encoder-decoder architecture to conditionally discriminates between real and generated speech
features.
The discriminator is used in a training pipeline that improves both the acoustic and prosodic features of a
TTS
model. We introduce our novel adversarial training technique by applying it to a FastSpeech2 acoustic model
and
training on Libriheavy, a large multi-speaker dataset, for the task of zero-shot voice cloning. Our model
achieves
improvements over the baseline in terms of speech quality and speaker similarity.
1Now affiliated with Cartesia AI.
System Architecture
Demo
To demonstrate our system we have provided 100 synthesized audios from the LibriTTS-R test set (unseen
speakers). Since our focus is on lightweight speech synthesis we use FastSpeech2 as a baseline model (which can
generate speech much faster than real-time on a CPU). However, the quality of zero-shot voice cloning with
FastSpeech2 is quite poor. We improve over the baseline model by adding a scaled up Transformer
discriminator which is fully described in the paper. Given that the discriminator is only used during training,
the improved FastSpeech2 model runs without any extra inference cost and can generate high quality voice cloning
in real-time on a CPU.
In addition to providing synthesized audios for the baseline and proposed models we also provide the ground
truth audio and the reference audio for comparison. The reference speech (which is passed to the models speaker
encoder) is an audio sample of the target speaker saying a different text. This better matches the zero-shot
voice cloning scenario where audio of the target speaker saying the target text is generally not available.
Reference Speech
Ground Truth
Baseline
Proposed Model
Avg NISQA: 4.30
4.26
3.27
4.15
Avg Speaker Similarity: 1.00
0.57
0.48
0.55
Pitch STD (%): 34.00
35.19
25.10
33.55
1089_134686_000012_000000: On the wall of his bedroom hung an
illuminated
scroll, the certificate of his prefecture in the college of the sodality of the Blessed Virgin
Mary.
1188_133604_000012_000002: My first and principal reason was that
they
enforced
beyond all resistance, on any student who might attempt to copy them, this method of laying
portions
of
distinct hue side by side.
121_121726_000007_000003: Painful to hear.
121_127105_000013_000000: "Oh, how delicious!" cried one of the
women.
121_127105_000016_000008: I adjured him to write by the first post
and
to
agree
with us for an early hearing; then I asked him if the experience in question had been his own.
1320_122612_000048_000002: Can you see where he has put his rifle
or
his
bow?"
1320_122617_000027_000002: But the bear, instead of obeying,
maintained
the
seat it had taken, and growled:
1580_141083_000008_000005: I was absent rather more than an hour.
1580_141083_000016_000000: "Bannister was very much upset by the
incident.
1580_141083_000018_000001: I have a new writing table with a fine
surface
of
red leather.
1580_141083_000041_000005: Holmes halted and looked earnestly at
the
window.
1580_141083_000076_000000: "Yes."
1580_141083_000117_000000: "Whom do you suspect?"
1580_141084_000006_000000: It was the Indian, whose dark silhouette
appeared
suddenly upon his blind.
1580_141084_000019_000001: The Indian still paced his room.
1995_1826_000016_000000: "What's the mere color of a human soul's
skin,"
she
had cried to a Wellesley audience and the audience had applauded with enthusiasm.
1995_1836_000004_000003: Why should he not be as other men?
1995_1836_000009_000000: mr Cresswell inclined his head
courteously.
1995_1837_000011_000001: Bles rushed to the edge of the swamp and
stood
there
irresolute.
2300_131720_000002_000003: There Messrs. Johnson and Hammer put
into
practice
many of the ideas now standard in the art, and secured much useful data for the work in New
York, of
which
the story has just been told.
2300_131720_000005_000002: For some years it was not found feasible
to
operate
motors on alternating current circuits, and that reason was often urged against it seriously.
2300_131720_000016_000014: But the plant ran, and it was the first
three
wire
station in this country."
2300_131720_000035_000004: The ladies thanked me effusively and
brought
their
visit to a close.
237_126133_000002_000003: At night, when no one knew it, the tears
would
come
racing over the poor, forlorn little face, and would not be squeezed back.
237_126133_000009_000000: "My son," said his mamma, "just stop and
think
how
badly you would feel, if you really couldn't see Percy."
237_134493_000017_000014: You must dance with all the French girls.
237_134493_000021_000002: A stranger, approaching it, could not
help
noticing
the beauty and fruitfulness of the outlying fields.
237_134500_000005_000000: "Don't let me disturb you, Emil.
260_123286_000006_000002: And yet, what cause was there for anger?
260_123286_000012_000001: I am not complaining that the rate is
slow,
but
that
the sea is so wide."
260_123286_000048_000002: The remainder of the sea monsters have
disappeared.
2830_3979_000023_000000: Glory to God in the highest, And on earth
peace,
Good
will to men.
2830_3980_000017_000000: When men claiming such credentials come
along,
they
deceive not only the naive, but also those who seemingly are well established in the faith. This
same
argument is used by the papacy.
2830_3980_000106_000004: He puts on white to make himself look like
an
angel of
light.
2961_961_000004_000008: The subject was a very noble one; he
described
the
most
famous action in which the Athenian people were ever engaged.
3570_5694_000010_000001: He is no longer simply the successful,
aggressive
male,--the man of strength, resource, and intrepidity.
3570_5694_000010_000004: This cultivation of aesthetic faculty
requires
time
and application, and the demands made upon the gentleman in this direction therefore tend to
change
his
life
of leisure into a more or less arduous application to the business of learning how to live a
life of
ostensible leisure in a becoming way.
3570_5695_000008_000002: The result is that, in order to keep up a
decent
appearance, the former habitually live hand to mouth to a greater extent than the latter.
3729_6852_000004_000003: In order to please her, I spoke to her of
the
Abbe
Conti, and I had occasion to quote two lines of that profound writer.
4077_13754_000002_000001: Then, in their hasty departure, the
soldiers
disposed
of everything outside of actual necessities in the way of accouterment and camp equipage.
4446_2271_000003_000003: But Hilda Burgoyne's the hit of the piece.
Hugh's
written a delightful part for her, and she's quite inexpressible. It's been on only two weeks,
and
I've
been
half a dozen times already. I happen to have MacConnell's box for tonight or there'd be no
chance of
our
getting places.
4446_2271_000017_000000: Mainhall cut in impatiently.
4446_2273_000011_000003: He painted that group of cypresses for the
Salon,
and
it was bought for the Luxembourg."
4446_2273_000023_000001: "Oh, yes, I go over to Paris often.
4446_2273_000027_000002: Too bad!
4446_2273_000034_000001: What a strong brown face she had, and very
tragic.
4446_2275_000021_000002: "Each life spoils the other.
4446_2275_000043_000004: It's getting the better of me.
4507_16021_000026_000001: Assuredly, if the tongue which a nation
or a
province
has spoken is worthy of interest, the language which has been spoken by a misery is still more
worthy of
attention and study.
4507_16021_000030_000001: Facts form one of these, and ideas the
other.
4507_16021_000038_000001: To diminish the number of the shady, to
augment
the
number of the luminous,--that is the object.
4970_29093_000017_000000: His pen therefore, and whereas, and not
as
aforesaid,
strayed off into other scribbling.
4970_29093_000017_000002: He would make his mark in literature.
4970_29093_000049_000001: He knew his uncle would be glad to hear
that
he
had
at last turned his thoughts to a practical matter.
4970_29095_000032_000001: "If thee wants to see new bonnets,
orthodox
to a
shade and conformed to the letter of the true form, thee must go to the Arch Street Meeting.
4992_23283_000047_000001: They are confused-they are triumphant at
present.
4992_41797_000012_000000: mr Popham laid down his brush.
4992_41797_000017_000002: She makes effort after effort, trembling
with
eagerness, and when she fails to reproduce what she sees, she works herself into a frenzy of
grief
and
disappointment."
4992_41797_000023_000002: But Nancy Carey, never shy, and as eager
to
give
herself as people always are who are born and bred in joy and love, Nancy hopped out of Mother
Carey's
warm
nest one day, and fixing her bright eyes and sunny, hopeful glance on the lonely, frowning
little
neighbor,
stretched out her hand in friendship.
5105_28241_000017_000000: The lieutenant paused to allow time for
reflection,
and added: "Moreover, I have every reason to believe that we are not so near the sun as we have
been.
5142_33396_000021_000000: "'Now for the farm.
5142_33396_000028_000001: It was so dark that I could see nothing
but a
few
sparks on the hearth.
5142_36600_000007_000007: The races differ also in constitution, in
acclimatisation and in liability to certain diseases.
5142_36600_000010_000001: All the more stress would be laid on this
fact,
as it
would be an exceptional one; for I am informed by mr Denny that the most different kinds of
dogs,
fowls,
and
pigeons, in England, are infested by the same species of Pediculi or lice.
5639_40744_000012_000003: For this and other reasons, Rodolfo
readily
complied
with the wishes of his father, who gave him ample letters of credit on Barcelona, Genoa, Rome,
and
Naples.
5639_40744_000016_000002: The bed she too well remembered was
there;
and,
above
all, the cabinet, on which had stood the image she had taken away, was still on the same spot.
5639_40744_000018_000002: But, as the proverb says, with the
disease
God
sends
the remedy.
5639_40744_000020_000002: The boy having embraced his mother,
calling
her
his
cousin, and his grandmother, calling her his benefactress, repeated his grandfather's question.
5683_32865_000031_000000: And Wylder laughed too, more suddenly and
noisily
than the humour of the joke seemed quite to call for, and glanced a grim look from the corners
of
his
eyes
on Lake, but the gallant captain did not seem to perceive it; and after a few seconds more he
handed
it
very
innocently back to mrs Dorothy, only remarking-
5683_32866_000018_000000: In the meantime, Miss Lake, with her hand
on
her
brother's arm, was walking swiftly under the trees of the back avenue towards that footpath
which,
through
wild copse and broken clumps near the park, emerges upon the still darker road which passes
along
the
wooded
glen by the mills, and skirts the little paling of the recluse lady's garden.
6829_68769_000005_000004: So this boy was doubly foolish in ruining
himself
to
get sixty dollars to pay an unjust demand."
6829_68771_000004_000000: BETH MEETS A REBUFF
6829_68771_000012_000005: The description she gave of the coming
reception
to
the Woman's Political League was so humorous and diverting that they were both laughing heartily
over
the
thing when the young people passed them, and thus mr Hopkins failed to notice who the occupants
of
the
other
vehicle were.
6829_68771_000037_000001: "Why cannot you answer it?"
6829_68771_000045_000000: "I wish," said Beth, slowly, "I knew
you."
6930_75918_000038_000000: "Yes, you are right; but I do not think
any
danger is
to be apprehended from Buckingham."
6930_75918_000039_000000: "No; still he is intrusive.
7021_79759_000004_000004: You have made a life-long change, if not
in
the
very
structure, at least in the permanent furnishing of her mind, and performed a work that can never
by
any
possibility be undone.
7021_85628_000014_000000: "Yes, why not?" thought Anders.
7127_75946_000052_000000: "Here?" replied the king, frowning,
"here?
7127_75946_000052_000001: Are you sure?"
7127_75946_000057_000002: I did not wish to be the substance of so
dark
a
shadow to your majesty's elegance, skill, and graceful invention; and I have left my tenants in
order to
place my services at your majesty's commands."
7176_88083_000024_000001: Then he came up among some arrowweeds,
lifted
his
head beneath the shelter of one of the broad barbed leaves, and floated there quivering.
7729_102255_000002_000001: The bogus Legislature numbered thirty
six
members.
7729_102255_000020_000006: He ordered the forces to disband;
prisoners
were
liberated, and with the opportune aid of a furious rain storm the Border Ruffian army gradually
melted
away.
8224_274384_000026_000002: And though the contributions which they
had
levied,
as well as the price of their living at free quarters, must be deducted, yet still the sum which
they
insisted on was very considerable.
8230_279154_000008_000005: For their mere occurrence, by itself,
would
not
suggest any connection with anything that had happened before.
8230_279154_000027_000001: This may be taken in two senses, the
first
when
a
thing not merely feels familiar, but we know it is such and such.
8455_210777_000054_000001: "It is simply a contest between brute
strength
and
mental energy."
8455_210777_000061_000003: Minister of the day thought it expedient
to
grant
their request.
8463_287645_000013_000000: "Another case was that of a slave woman
in a
very
delicate state, who was one day knocked down stairs by mrs Johnson herself, and in a few weeks
after,
the
poor woman died from the effects of the injury thus received.
8463_287645_000024_000001: I always had it in my mind to leave, but
I
was
'jubus', (dubious?) of starting.
8463_294828_000020_000000: For the past ten years, Conseil had gone
with me
wherever science beckoned.
8463_294828_000024_000002: There was good reason to stop and think,
even
for
the world's most emotionless man. What would Conseil say?
8463_294828_000052_000000: "In person.
8555_284449_000038_000001: The guards had a terrible struggle with
the
goat,
which was loose in the room and still wanted to fight, but finally they subdued the animal and
then
they
took the Boolooroo out of the frame he was tied in and brought both him and the goat before
Queen
Trot,
who
awaited them in the throne room of the palace.
8555_284449_000061_000000: "I think it will be best for us to go
back
to
our
own country as soon as possible," suggested Rosalie the Witch; "for, if we stay here very long,
the
Blueskins may rise against us and cause the Pinkies much trouble."
